JMRI: XML Schema Usage

JMRI uses XML for a number of purposes: to hold decoder definitions, for its persistance system for configuration and panel information, and to create parts of the web site from other files. This page describes how we specify the content of those files using XML schema.

For examples (not a tutorial!) on the structure of our schema, see the examples page.

The current schema can be seen online in the schema directory. The most commonly used one is the layout.xsd schema for panel files. See below on how it's organized.

Access to Schema Definitions

JMRI uses XML Schema to define the format of its files.

Those XML Schema may need to be available to the program when it reads the files, as they define the default values of missing attributes and other needed information.

In the JMRI distributions, these are stored in the xml/schema directory. Note that they are not stored in each directory alongside the XML files. There are just too many locations to keep such a set of schema definition files up to date. JMRI itself, via the jmri.jmrit.XmlFile class, provides support for locating those files when the XML parser needs them.

Developing JMRI Schema

Our preferred organization for XML schema is based on the structure of the underlying code: A particular *Xml class is the unit of reuse.

Lots of classes descend from jmri.configurexml.XmAdapter: (see JavaDoc)

By convention, provide <xsd:annotation><xsd:appinfo> element containing the class name that reads/writes the element:


    <xs:annotation>
        <xs:documentation>
        Some human readable docs go here
        </xs:documentation>
        <xs:appinfo>
            <jmri:usingclass configurexml="false">jmri.managers.DefaultSignalSystemManager</jmri:usingclass>
        </xs:appinfo>
    </xs:annotation>

The Venetian Blind Pattern

We are moving toward structuring our XML using the "Venetian Blind pattern". In this style, the top level elements that are written by classes have types defined for them. Any elements that are within those are defined anonymously, within those elements. For an example of this, see the types/sensors.xsd file, which defines a type for the "sensors" element written for SensorManagers. Within that, there is included a definition of a "Sensor" element, and a "comment" element within that.

This limits the number of types, and keeps the schema files roughly aligned with the classes that do the reading and writing.

There are a few items (elements and attribute groups) that extend across multiple types. They are defined in the types/general.xsd file.

More information on XML schema design patterns is available at DeveloperWorks. developers.sun, and javapassion.

Available Defined Types

systemNameType: System names, to eventually be tightened up to a real test of validity
userNameType: User names, not including the empty name
nullUserNameType: User names, with an empty value allowed
beanNameType: Either user or system name
turnoutStateType: closed, thrown
signalColorType: red, yellow, etc
yesNoType: yes, no
trueFalseType: true, false

Schema Versioning

The requirements for versioning of XML Schema are much simpler than those for DTDs, mostly because individual items have much clearer scope.

You do need to version a schema you make a change to the schema such that previous files will no longer validate with the current schema.
In that case, just make your schema changes in the current schema document, and commit them back to CVS.
You do not need to version the schema if you add or change things such that existing files continue to validate.
In this case, the steps to version the schema are:
1. Copy the current schema file to a new one with the next version number. E.g. copy types/turnouts-2-9-6.xsd to types/turnouts-2-9-8.xsd. Make your changes and commit that new version.
2. If this is a subfile, such as the types/turnouts-2-9-6.xsd, that is included in a main schema such as layout-2-9-6.xsd, that main file also needs to be copied, have the include changed, and commited to CVS.
3. Then, change the Java code that writes the schema reference to the top of output files to use the new filename. For example, layout (panel) files are written by src/jmri/configurexml/ConfigXmlManager.java.
4. If the XML stylesheet(s) in xml/XSLT have to be changed, they are versioned in a similar way, including the code change where the stylesheet reference is included.

In either case, it's important to include sufficient test files that the unit tests catch any problems with the new and old schema. See the test section below.

Checking JMRI Schema

XML Schema are more verbose than the DTDs we're used to. With that complexity, we get more expressing power, but at the same time more ways to mess up.

It's important that the JMRI schema definitions be kept semantically correct. There are some subtle aspects to XML schema that it will take us some time to get comfortable with, and if we let too many problems build up in the meantime, we'll eventually have a lot of back-fitting to do. The W3C online schema validation tool is a very good tool for checking that JMRI schema changes are still technically correct. You should check your changes with it before commiting them to CVS. Unfortunately, it doesn't seem to to check compliance with nested schema elements, e.g. from DocBook (see below) or JMRIschema, but it's still a very useful check.

Using the JMRI "Validate XML File" tool in the "Debug" menu to validate a .xml file ("instance file") that uses your new or updated schema is an important check of both.

You should also add a JUnit test that checks the schema and a typical file. The easiest way to do that (see e.g. test/jmri/configurexml/SchemaTest.java for an example) is to inherit your test class from jmri.configurexml.LoadFileTestBase, and then call validate(File f) on a test file. The test XML file can put put with the test Java file in the "test/" file tree.

    public void testValidateRoster() {
        validate(new java.io.File("java/test/jmri/configurexml/RosterSchemaTest.xml"));
    }

For a quick file check, Linux and MacOS X users can validate from the command line with e.g.

 cd xml
 xmllint -schema schema/aspecttable.xsd -noout signals/sample-aspects.xml

xmllint can't check schema files themselves, unfortunately, because their schema isn't something it can handle.

Your schema docs should point to our standard stylesheet in their head matter:

<?xml-stylesheet href="schema2xhtml.xsl" type="text/xsl"?>

This is a pretty basic stylesheet, only really sufficient to show basic structure. For an example of the output it produces, click on this link to the aspecttable.xsd schema file. If anybody knows of a better stylesheet, we can certainly switch to it.

JUnit testing

We test XML schema and files two ways: Do they properly validate, and can certain files be properly read in and then written back. The second test is more complicated, of course, but also more valuable.

To test for validation, drop an XML file into the test/jmri/configurexml/files directory. This will automatically be tested to make sure it validates. If you version a schema, you should run a few of these files through PanelPro (e.g. read them and then rewrite them with the new format) and include new copies to test the new format.

Small fragments that demonstrate specific schema features can be put in the test/jmri/configurexml/pass and test/jmri/configurexml/fail directories (the "fail" directory files are expected to fail for some specific reason documented via comments in the file).

To read and write, see the example in test/jmri/configurexml/LoadFileTest.java

External Standards and Future Work

The OASIS collaboration defines a number of schema and schema elements that have become well-known standards. Were possible, we should use those standard elements to improve inter-operability. The first ones of interest are:

DockBook defines elements for several concepts we use:
- author (http://www.docbook.org/tdg/en/html/author.html)
- address (http://www.docbook.org/tdg/en/html/address.html)
- revision history (http://www.docbook.org/tdg/en/html/revhistory.html)
See
We have our own DocBook subset that we use, because the full DocBook 5.0 schema takes a very long time to parse, and isn't fully consistent with versions of other software that we use. We use the normal DocBook 5.0 namespace, so we can later easily convert to a more complete schema transparently. Our smaller schema is located at http://jmri.org/xml/schema/docbook/DocBook.xsd (our usual schema location). It is only referenced from JMRI schema files, not instance files, so that we can later convert with finite work.
UBL, though aimed at business transactions, defines elements to represent parties (companies, people), devices, model numbers, etc.
OpenDocument (OODF) defines a set of elements and structures for computations as part of its spreadsheet module. (But they provide Relax-NG schema, not W3C XML Schema, so that doesn't help so much)

Learning to use these will require some work, as we can't assume that computers using JMRI have internet access, so can't just reference the entire schema as remote entities.

Copyright, Author and Revision Information

For various reasons, we need to move to DocBook format for Copyright, Author and Revision information in our XML files (instance files).

Sample XML:

  <db:copyright>
        <db:year>2009</db:year>
        <db:year>2010</db:year><
        db:holder>JMRI</db:holder></db:copyright>

  <db:authorgroup>
    <db:author>
        <db:personname><db:firstname>Sample</db:firstname><db:surname>Name</db:surname></db:personname>
        <db:email>name@com.domain</db:email>
    </db:author>    
  </db:authorgroup>

  <db:revhistory>
    <db:revision>
        <db:revnumber>1</db:revnumber>
        <db:date>2009-12-28</db:date>
        <db:authorinitials>initials</db:authorinitials>
    </db:revision>    
  </db:revhistory>

Sample schema description: (But see the real one, which is provided in schema/docbook)

    <xs:element ref="docbook:copyright" minOccurs="1" maxOccurs="1" >
      <xs:annotation><xs:documentation>
      DocBook element(s) providing copyright information in standard form.
      Must be present.
      </xs:documentation></xs:annotation>
    </xs:element>

    <xs:element ref="docbook:authorgroup" minOccurs="1" maxOccurs="unbounded" >
      <xs:annotation><xs:documentation>
      DocBook element(s) describing the authors in standard form
      </xs:documentation></xs:annotation>
    </xs:element>

    <xs:element ref="docbook:revhistory" minOccurs="1" maxOccurs="unbounded" >
      <xs:annotation><xs:documentation>
      DocBook element(s) describing the revision history in standard form
      </xs:documentation></xs:annotation>
    </xs:element>