Future Data Exchange Standard

From Evolutionary Informatics Working Group
(Redirected from NeXML)
Jump to: navigation, search

This page covers NeXML.

Introduction

In the first evoinfo meeting the working group acknowledged the desirability of a formal ontology as the basis for data interoperability, and has chosen the development of a General Ontology as one of its goals. Such a formal ontology, for example in RDF/XML, can be translated directly into a serialization protocol and into a file standard definition such as an XML Schema. However, the requirements of this ontology were not clearly established, and a serialization protocol for it would not automatically result in a convenient next generation file standard. Hence, some attendants argued that a more "bottom up" approach that starts with the design of a new standard will help uncover the hidden requirements of the ontology and generate the serialization protocol in the process. This page describes recent discussion and ongoing work on the design of this new standard.

Goals for the working group

At the first meeting (20-23 May, 2007), the working group decided that work on the future data exchange standard would primarily proceed under the auspices of Rutger Vos who has reported on its progress at the Fall '07 meeting, and further reporting happened at the third meeting.

RFC: nexml

In internetworking and computer network engineering, Request for Comments (RFC) documents are a series of memoranda encompassing new research, innovations, and methodologies applicable to Internet technologies. RFCs are meant for peer review and to transmit new concepts, and are published through the Internet Society. This document is therefore in strict terms not a real RFC, but it serves the same purpose: to provoke comment and discussion, and to disseminate information about the design process of a new data exchange standard for phylogenetics, which in honor of the commonly used nexus flat file format is called nexml. The status of this page is neither that of a formal specification or recommendation, nor formally endorsed by evoinfo in its current stage.

Preamble

The following subsections introduce the context within which nexml development takes place, describing the interoperability problems faced by phylogeneticists, the requirements nexml needs to meet to address these problems, and the development approach that is being taken to meet these requirements.

Pressures

Current practices and ongoing developments in phylogenetics add pressure to the need for a new data standard. The working group is compiling a list of these issues, some of which include:

  • Many flat file formats and dialects: in recent years, phylogeneticists have adopted a number of different file formats to store data such as trees and sequence alignments. Unfortunately, most flat file formats used for this purpose generally lack an exhaustive, formal specification - and even if they do, they have been extended in various ways. Examples of this abounded during the meeting; for instance:
    • pipe-delimited meta-data fields in fasta files
    • mutually exclusive syntax for polymorphy in nexus matrices: {a,c} vs. a&c
    • "hot comments" in tree descriptions
    • "mixed" data in mrbayes nexus matrices
    • proprietary substitution model descriptions
    • extensions of molecular symbols list beyond IUPAC standard
  • Data hard or impossible to validate automatically: for many file formats, no validation procedure exists - and given the proliferation of mutually exclusive dialects it's difficult to see how such a procedure could deduce validity unambiguously. What might be valid "nexus" to one program, is rejected by another program. This makes many types of compatibility issues hard to solve without extensive knowledge of file conventions "in the wild".
  • Bigger data sets: meanwhile, phylogenetic analyses are continuously growing in size and complexity. Solving compatibility issues by hand is becoming too tedious.
  • Increased automation: the growth in size and complexity of analyses has promoted development of automated systems for data analysis (pipelines/workflows) and data storage (databases). The purpose of a workflow is obviously defeated if it requires manual editing of data files between steps in the analysis. As well, the reproducibility of results obtained from complex workflows is in doubt if no formal facility exists for logging the steps undertaken during an analysis.
  • More data types: since the introduction of the nexus file format, new types of analyses (e.g. Bayesian, ML) have gained ground, which have introduced new types of information (substitution models, priors and posteriors on branches) for which no standardized format exists.
  • Nexus problems: as the nexus file format is the most commonly used format in phylogenetics, the working group is compiling a separate list of nexus problems.

Requirements

To address the issues described in the preceding section, a new standard should meet at least the following requirements:

  • Formalization: the next standard implements a formal ontology and is defined by a schema (as in phyloxml), such that instance documents can be validated with great granularity, including for type safety for character data (per character), referential integrity between entities, validity of tree descriptions;
  • Extension: the next standard extends current standards to include model descriptions, reticulate trees, arbitrary attachments on nodes, trees, tree sets, sites, sequences, alignments, etc.;
  • Data integration: the next standard allows for the integration of data from disparate sources, e.g. from multiple files, from databases, from sets of alignments and trees, from existing ontologies;
  • Legacy compatibility: the next standard is organized conceptually, and can be serialized or otherwise represented in ways that are understandable for legacy software, such as being able to express the standard nexus subset;
  • Analysis context description: the next standard can represent meta-information such as analysis procedures, instance document changes, schema changes;
  • Parseability: the next standard is easily parsed without the need for custom tokenization, both in compact and verbose representations;

Approach

  • Development of the data exchange standard is now ongoing with input from various contributors, including, in alphabetical order, Jason Caravas, Mark Holder, David Maddison, Wayne Maddison, Peter Midford, Andrew Rambaut, Jeet Sukumaran and Rutger Vos as well as other members of the evoinfo group and attendants of the pPOD kickoff meeting.
  • The svn repository has migrated to sourceforge. It contains a modularized XML schema (discussed on powerpoint slides) and various instance documents demonstrating representations of common phylogenetic data.
  • Nexml parsers in perl, python, javascript, c++ and in java are under development. Here are some code samples for perl.
  • The domain name http://www.nexml.org has been registered, which serves to uniquely identify the nexml namespace and for which a web presence which aggregates various information streams (mailing list, wiki changes, svn commits, bugs) has been established.
  • NESCent has provided a staging server from which this web presence is launched on the production server.
  • To facilitate development of the ontology, concepts introduced by the design of the schema and discussed on this wiki page are cross referenced with the glossary.
  • An online validation service is under development. You can reuse it by placing this in an html document:

<xml>

<form action="http://www.nexml.org/nexml/validator" enctype="multipart/form-data" method="post">
  <input type="file" name="file"/>
  <input type="submit"/>
</form>

</xml> Or use it as a simple REST service: HTTP response code 201 means it's valid, 400 means it isn't. The validator goes through a two-step process:

  1. grammar-based validation: this is the part where we check an input file for sensible placement of elements, attributes and text nodes. We do this by validating against the xml schema using a Xerces-J validator writting by Terri Schwartz of the CIPRES-SDSC team. Unfortunately, some constraints we like to place on files cannot be expressed in schema language. For this we have to resort to the second step.
  2. rule-based validation: here we test whether chained remote references (rules such as "matrix row X is part of characters block Y, which is linked to OTUs block Z. Therefore, X can only refer to an OTU that is part of Z") make sense. We do this by parsing the file using Bio::Phylo's nexml parser.

Design


The following subsections describe the design of the developing nexml standard. These descriptions are likely to both be in flux, and lagging behind the formal (one, true) XSD description in the svn repository. The reader is therefore advised to follow the links in the subsections to the XSD fragments and XML instance document examples that define the elements.

General design

Verbosity

XML in general is usually far more verbose than flat file formats that convey the same information. Nexml is designed to allow both for verbose and compact representation of data. The rationale is that some use cases (e.g. submission of morphological observations to a database) require extensive annotation, whereas others (e.g. submission to a processing engine as part of a workflow) don't require a lot of metadata, only the bare bones needed to complete the analysis step.

References

Elements other than xml structural placeholders (e.g. the matrix element that lumps matrix rows together) all have the following properties:

  • a required id attribute. This attribute is not a standard XML id (which are file scoped), but of type xs:NCName, which have the same string format restrictions but are free to have their scope defined by the schema. In the nexml schema, all such ids are scoped within their enclosing elements. This means that nodes must have an id that is unique within the tree that contains them, matrix rows must have ids unique within the matrix, etc. The rationale is that this minimizes the risk of clashes when combining data from multiple sources, and makes it easier to solve clashes should they occur (by re-assigning the id of the enclosing element).
  • some elements must refer to other elements: e.g. a matrix row element must refer to a otu element. By convention, the attribute used to define the id of another element is named after that referenced element. For example, <row otu="t1" id="r1"/> means that the row element identified by id r1 links to a otu element whose id is t1. Referenced elements in nexml always precede their references, which is why otus elements come before characters elements, for example.
Metadata

To allow for the preservation of arbitrary labels and annotations, nexml elements have:

  • an optional label attribute, which is used to specify a human-readable label for the element. The only restriction on this attribute's value is that it is a valid xml string, which means that reserved xml tokens (such as the greater-than symbol >) must be escaped using xml entities.
  • zero or more <dict/> elements (dictionary attachments) can be enclosed within them. These attachments, if present, precede any other enclosed elements (the rationale is similar to that for having referenced elements precede their references: it facilitates stream based parsing strategies).
Other common attributes

In addition to the shared attributes and structures described above, nexml elements also optionally have:

  • a class attribute. This attribute has an id reference to a class defined earlier as its value. Its function is analogous to that of sets (e.g. OTU sets, character sets) in nexus, but instead of the set declaring what it contains here the nexml elements declare what set(s) they belong to. This is the idiomatic approach for XML, and is compatible with CSS styling.
  • an xml:base attribute, from the core XML namespace. This attribute has an absolute URL as value, which is the base address upon which relative links defined later in a document are built. There is a similar facility in HTML (defined in the head of a document), which in practice is used to shorten URLs in pages by defining their shared path fragments in one location. The rationale for inclusion in nexml is similar, with the additional consideration that some XInclude processors place the attribute by default in the post-include infoset.
  • an xlink:href attribute, from the XLink namespace. This attribute has an absolute or relative URL as its value, which is used to create a unidirectional link from the element that has the attribute to another location.
  • an xml:id attribute, from the core XML namespace. This attribute has a string of the same format as the nexml id as its value, except this id is file-scope unique. The attribute has no defined function within nexml, but some XInclude processors might include it.
  • an xml:lang attribute, from the core XML namespace. This attribute has an RFC3066 compliant language code (e.g. en-US) as its value, which is used to indicate the natural language in which text of the element or its children are written.
  • In addition, attributes from any other namespace are allowed (i.e. foo:bar="baz"), provided this namespace (say, http://foo.org) is declared previously and bound to a namespace prefix (foo:), i.e. xmlns:foo="http://foo.org" foo:bar="baz"

Element description


Root

[schema], [example]


(Like all standalone xml files, nexml files start with the <?xml version="1.0" encoding="some character encoding scheme"?> processing instruction. The encoding scheme is used to specify what character set - e.g. some unicode flavor - is used. Hence, nexml files can include OTU descriptions in simplified Chinese, for example, without confusing parsers. Naturally, i18n in terms of character sets is best used in combination with xml:lang attributes where appropriate.)

The root element of the schema is called nexml. This root element has the following attributes:

  • a required version attribute whose value is a decimal number indicating the nexml schema version. At present this value is 0.8, with the 1.0 release version slated for Spring 2009.
  • an optional generator attribute, which is used to identify the program that generated the file. The attribute's value is a free form string.

In addition, it will commonly define a number of xml namespace prefixes. (Where it says "by convention" in the list below, the convention applies to the three-letter prefixes which are free to vary in most cases, not the namespaces themselves):

  • the xml namespace prefix that identifies xml schema semantics that might be inlined in the file. By convention this is of the format xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" so that parts of schema language used inside nexml (e.g. where a concrete subclass must be specified) are identified by the xsi prefix.
  • the nexml namespace prefix, by convention of the format xmlns:nex="http://www.nexml.org/1.0", so that locations where nexml specific types are referenced (e.g. data type subclasses) these are identified by their nex prefix.
  • the default namespace, xmlns="http://www.nexml.org/1.0", which is necessary for namespace aware processors (such as the nexml=>cdao xslt stylesheet produced by a subgroup at the Spring 2009 hackathon).
  • the xml namespace prefix, required to be of the format xmlns:xml="http://www.w3.org/XML/1998/namespace". This may be used, for example, to specify the base address of imported nexml snippets (using the xml:base="http://example.com/base.xml" attribute) and to specify the language in which certain element contents are written (using, say, xml:lang="nl").

Lastly, to associate the instance document with the nexml schema, it requires an attribute to specify the nexml schema location, and the namespace it applies to. This is of the format xsi:schemaLocation="http://www.nexml.org/1.0 nexml.xsd" (in a stable release the location of the schema would not be a local path - such as nexml.xsd - but a url). Notice that this attribute is a schema language snippet (identified by the xsi: prefix) that identifies a namespace (http://www.nexml.org/1.0) and associates it with a physical schema location (../nexml.xsd).


Together, this makes the root element look something like the following: <xml>

<?xml version="1.0" encoding="ISO-8859-1"?>
<nex:nexml
    version="0.8"
    generator="eclipse"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:xml="http://www.w3.org/XML/1998/namespace"
    xmlns:nex="http://www.nexml.org/1.0"
    xmlns="http://www.nexml.org/1.0"
    xsi:schemaLocation="http://www.nexml.org/1.0 http://www.nexml.org/1.0/nexml.xsd">
</nex:nexml>

</xml> The root element can hold:

  • zero or more attachment dictionaries,
  • one or more OTUs elements,
  • zero or more characters elements,
  • zero or more trees elements (in mixed order with characters elements),
OTUs

[schema], [example], [term],


In some phylogenetic analyses (such as Bayesian analyses that yield a set of trees) many nodes [term] - in different trees [term] - refer to the same sequence. On the other hand, in some other analyses (such as those that involve simulation of a set of sequence alignments) many sequences - in different alignments - refer to the same node in the generating tree. This relationship can be normalized by creating a third entity from which one-to-many links point both to nodes and sequences. In nexus files, these entities are defined in the taxa block using a set of labels that sequences and nodes later on in the file must refer to. In nexml, this same functionality is provided by otus elements. The name change is a result of the progressing integration between phylogenetics and taxonomy [term], which now causes concept confusion because of the overloading of the term "taxa" when it was introduced in the nexus standard.

In its simplest form, a otus element looks something like this: <xml>

   <otus id="tax1">
       <otu id="t1"/>
       <otu id="t2"/>
   </otus>

</xml> That is, the otus element and its contained otu elements require a (file level unique) id attribute. In addition, these elements can have an optional label attribute that defines a human readable name for the element, and can contain dictionary attachments.

Characters

[schema], [example], [term]


The characters element is somewhat analogous to the nexus characters blocks: it stores data such as molecular sequences, categorical data or continuous data. The element is different from the nexus characters block in that it allows for more detailed specification of the allowed states per character, strict validation of the observed states, annotation of characters (columns), states, rows and individual observations using dictionaries. In addition, the characters element is designed to allow for representation of non-homologized data: the element is more accurately described as a bucket of observations and the allowed parameter space for those observations. Only if the boolean attribute aligned of the matrix element is set to "1" (true), can subsequent observations be assumed to be homologous across row elements.

The schema specifies the characters element to be of an abstract type, so that instance documents need to specify the concrete subclass (i.e. datatype) using the xsi:type attribute. At present, six data types are supported: DNA, protein, restriction sites, standard categorical, continuous and RNA. For each of these data types there are two subclasses, whose names are constructed as "data type"Seqs (e.g. DnaSeqs) and "data type"Cells (e.g. DnaCells), the former being a compact representation with states listed as tokens in a sequence, the latter a verbose representation with states marked up granularly.

STANDARD

For categorical data (e.g. xsi:type="nex:StandardCells"), the row elements inside the matrix element must be preceded by a format element that specifies the allowed states [term] per character [term]. An example is shown below: <xml>

       <format>
           <states id="states1">
               <state id="s1" symbol="1"/>
               <state id="s2" symbol="2"/>
               <state id="s3" symbol="3"/>
               <state id="s4" symbol="4"/>
               <polymorphic_state_set id="s5" symbol="5">
                   <mapping state="s1"/>
                   <mapping state="s2"/>
               </polymorphic_state_set>
               <uncertain_state_set id="s6" symbol="6">
                   <mapping state="s3"/>
                   <mapping state="s4"/>
               </uncertain_state_set>
           </states>
           <char states="states1" id="c1"/>
           <char states="states1" id="c2"/>
       </format>

</xml> In this case, then, the matrix holds two six-state characters. State s5 functions as an ambiguity code, a state that actually means a set of two other states (s1 and s2). Because the element is called polymorphic_state_set, the mapping to the two other states indicates true polymorphism (i.e. both states are observed in a population). State s6 is uncertain. In practice, "polymorphism" can be read as "AND", and "uncertain" as "OR". states, polymorphic_state_set, uncertain_state_set,state and char elements can take dictionary attachments, which are loosely analogous to - but more powerful than - the nexus tokens CHARSTATELABELS and CHARLABELS, respectively.

Subsequent row elements contain the mappings between the defined states and the actual observations. For example: <xml>

   <matrix>
           <row id="r1" otu="t1">
               <cell char="c1" state="s2"/>
               <cell char="c2" state="s2"/>
           </row>
   </matrix>

</xml> This structure means that the entity defined by OTU "t2" has state "s2" (symbol 2) for both characters. Therefore, cell elements within different row elements are homologous [term] if they have the same value for the char attribute. DNA, RNA, protein and restriction data can be described in a similar, verbose way, with the "symbol" attribute's value being an IUPAC nucleotide symbol, an IUPAC amino acid symbol or a boolean (0/1), respectively.

In a compact representation, the same STANDARD information is marked up like this: <xml>

<row id="r2" otu="t2"><seq>2 2</seq></row>

</xml> Notice how the symbols are space-separated (this is because STANDARD states aren't necessarily single-character symbols: integers greater than 9 are allowed also).

DNA, RNA, Protein and Restriction

In its most compact form, a DNA sequence alignment as expressed in a characters element would look something like this: <xml>

<characters otus="tax1" id="m1" xsi:type="nex:DnaSeqs">
    <format>
    </format>
    <matrix aligned="1">
        <row id="r1" otu="t1"><seq>AACATATCTC</seq></row>
        <row id="r2" otu="t2"><seq>ATACCAGCAT</seq></row>
        <row id="r3" otu="t3"><seq>GAGGGTATGG</seq></row>
        <row id="r4" otu="t4"><seq>GGTCTTAGAG</seq></row>
        <row id="r5" otu="t5"><seq>CGTCACAGTG</seq></row>
    </matrix>
</characters>

</xml> The example above shows how for a compact representation of DNA characters are concatenated as a string, because all the symbols are single characters. The RNA subclasses are virtually identical to the equivalent DNA subclasses (only T vs U). The protein and restriction data types also both use single character symbols, whose allowed symbols are well-defined, namely as the IUPAC single character amino acid symbols and as 0 or 1, respectively.

CONTINUOUS

For continuous data, the format element defines the characters (i.e. char elements) but not their states, and observations values (i.e. either the state attribute in verbose notation, or space-separated symbols in compact notation) are arbitrary precision floating point numbers.

Combining data

In some analyses, data of different types is analyzed jointly (e.g. mrbayes does this). Nexml does not currently facilitate this. It is likely that this will be implemented either using matrix sets or (less likely) using a MIXED concrete subclass.

Trees

[schema], [example]


Due to their nesting, tree descriptions as nested elements (as suggested in Joe Felsenstein's book Inferring Phylogenies) can pose special problems for xml parsers: a parser can only hand off an element once all its children have been processed and stored in memory. Large trees described using nested elements can therefore develop huge memory requirements. Hence, nexml describes trees as node and edge [term] tables instead, following the syntax for GraphML (which is also discussed separately in the context of the study of related artefacts).

Tree

[term]

The concrete subclasses IntTree and FloatTree describe a tree shape following GraphML syntax. The classes differ in that the optional length attribute is either an integer or a IEEE 754-1985 compliant floating point number. Below is an example: <xml>

<tree id="tree1" xsi:type="nex:FloatTree" label="tree1">
    <node id="n1" label="n1" root="true"/> 
    <node id="n2" label="n2" otu="t1"/> 
    <node id="n3" label="n3"/>
    <node id="n4" label="n4"/>
    <node id="n5" label="n5" otu="t3"/>
    <node id="n6" label="n6" otu="t2"/>
    <node id="n7" label="n7"/>
    <node id="n8" label="n8" otu="t5"/>
    <node id="n9" label="n9" otu="t4"/>
    <rootedge target="n1" id="re1" length="0.34765" />
    <edge source="n1" target="n3" id="e1" length="0.34534"/>
    <edge source="n1" target="n2" id="e2" length="0.4353"/>
    <edge source="n3" target="n4" id="e3" length="0.324"/>
    <edge source="n3" target="n7" id="e4" length="0.3247"/>
    <edge source="n4" target="n5" id="e5" length="0.234"/>
    <edge source="n4" target="n6" id="e6" length="0.3243"/>
    <edge source="n7" target="n8" id="e7" length="0.32443"/>
    <edge source="n7" target="n9" id="e8" length="0.2342"/>
</tree>

</xml> This is an XML representation of the newick string (((t4,t5)n7,(t2,t3)n4)n3,t1)n1;. In this representation, the root [term] is principally identified by having in-degree of zero or one, i.e. no edge element exists with a target attribute that references that node, but a rootedge element may exist to indicate a time span leading up to the root (principally for coalescent trees). An additional root attribute is used to indicate that this tree is in fact considered truly rooted. This attribute may be used on multiple nodes, to indicate multiple rootings. Tips [term] are identified by there being no edge elements with source attributes that reference them.

To add additional objects to nodes or edges, such as bootstrap [term] values, a dictionary attachment is used.

Network

[term]

The IntNetwork and FloatNetwork subclasses only differ from the tree subclasses in that the key constraints on the in-degree of nodes is lessened, so that a node can have multiple parents. In the example below, node n6 has an additional parent node n7, creating a reticulation: <xml>

<tree id="tree3" xsi:type="nex:IntNetwork" label="tree2">
    <node id="n1" label="n1"/>
    <node id="n2" label="n2" otu="t1"/>
    <node id="n3" label="n3"/>
    <node id="n4" label="n4"/>
    <node id="n5" label="n5" otu="t3"/>
    <node id="n6" label="n6" otu="t2"/>
    <node id="n7" label="n7"/>
    <node id="n8" label="n8" otu="t5"/>
    <node id="n9" label="n9" otu="t4"/>
    <edge source="n1" target="n3" id="e1" length="1"/>
    <edge source="n1" target="n2" id="e2" length="2"/>
    <edge source="n3" target="n4" id="e3" length="3"/>
    <edge source="n3" target="n7" id="e4" length="1"/>
    <edge source="n4" target="n5" id="e5" length="2"/>
    <edge source="n4" target="n6" id="e6" length="1"/>
    <edge source="n7" target="n6" id="e7" length="1"/> 
    <edge source="n7" target="n8" id="e7" length="1"/>
    <edge source="n7" target="n9" id="e8" length="1"/>
</tree>

</xml>

Sets

Nexus files can define sets of various things. In NeXML this is implemented using class class elements and attributes. This is most easily demonstrated with an example:

<xml>

   <otus id="tax1">
       <dict id="dict1">
           <string id="description">example of nexml classes</string>
       </dict>
       <class id="class1" label="Apes"/>
       <class id="class2" label="Lemurs"/>
       <class id="class3" label="Primates">
           <dict id="dict2">
               <string id="description">here's how to annotate a set of things</string>
           </dict>
       </class>
       <otu id="t1" class="class1 class3" label="Pan paniscus"/>
       <otu id="t2" class="class1 class3" label="Gorilla gorilla"/>
       <otu id="t3" class="class2 class3" label="Eulemur mongoz"/>
       <otu id="t4" class="class2 class3" label="Lemur catta"/>
       <otu id="t5" class="class3" label="Rhinopithecus roxellana"/>
   </otus>

</xml> Note: this feature hasn't been fully implemented in the schema, so validation of files containing class elements might fail.

Dictionaries

[schema]


Future developments in phylogenetics are impossible to predict - new support values for nodes on trees may emerge, new types of annotations for DNA sequences, etc. In addition, the amount and types of metadata that can be attached to phylogenetic data are unlimited: some researchers may want to attach taxonomy database ids to otu elements, or genbank accession numbers to DNA and so on. A data exchange standard that attempts to limit the universe of "stuff" to attach to "things" is likely to be headed for immediate obsolescence. The nexml standard therefore allows for attachment of arbitrary key/value pairs to:

  • the root element,
  • otus and otu elements,
  • the characters element and its character definitions, state definitions, matrix rows, sequences and individual observations,
  • the trees, individual tree elements and nodes.

Or, in general, any element that inherits from the Annotated abstract type (i.e. everything other than placeholder elements). Here is a simple example: <xml>

<dict id="dict1">
    <string id="description">This is a string based description of an element</string>
</dict>

</xml> A dict element contains a sequence of key/value pairs, where the id attribute contains a string (scoped to be unique within the enclosing dictionary), and the value can take on a number of different types:

  • integer or integervector
  • float or floatvector
  • string or stringvector
  • boolean or booleanvector
  • id or idvector
  • date or datevector, i.e. a restricted date/time format
  • url or urlvector
  • base64, i.e. for base64 encoded binary data such as images;
  • dict, i.e. another dictionary, yielding a recursive data structure;
  • any, i.e. XML from another namespace such as XHTML (for marked up text), SVG (for graphs), ant build scripts, etc. This feature is used extensively in a discussion of ontology-mediated metadata attachments.'

The implication is that dictionaries map onto data structures much like hashtables, python dictionaries, or perl hashes but with type-safe values. In principle, this structure thus allows all kinds of things to be attached to elements, the only downside being that different researchers might use different substructures or types. For example: <xml>

<dict><string id="genbank">NM_052988.2</string></dict>

</xml> and: <xml>

<dict><string id="accession">NM_052988.2</string></dict>

</xml> would not be recognized to mean the same thing without human intervention. The solution is to create attachments of type any whose value is RDF/XML so that reasoners can figure out, with recourse to external ontologies, that the same thing is meant. This is discussed in detail elsewhere on this wiki.

Resources

Here's a slide show about NeXML, on google presentation: <embedurl>http://docs.google.com/Present?docid=dc3k7mhr_8g9j2j7ct</embedurl>

SourceForge

Useful sections of the nexml website

  • On every page of the website, at the top of the page, is a form where you can:
    • Validate NeXML files. This is useful for developers to make sure they're emitting legal NeXML.
    • Translate Nexus to NeXML. This is useful for generating templates.
    • Translate NeXML (or any xml) to JSON. Not much use for now, actually - but could be nice for web apps.
  • example files
  • schema documentation
  • downloads

RSS Feeds

Here are some useful RSS feeds to monitor:

XML Parsers

Below are the most commonly used open source XML parsers. Which should I use? Apache and MIT licenses are both compatible with closed source projects, i.e. no copyleft requirement. SAX is the stream-based api (more memory-efficient, but need to keep track of context yourself), DOM is the tree-based api (easier to traverse, harder on memory). Expat is also used under the hood by perl xml parser XML::Twig. It's (supposed to be) the fastest xml parser. Gnome is sometimes hard to build on Windows, so I'd use expat for SAX or xerces for DOM. For java, the nexml class libraries use whatever lives under org.xml.sax.*

  • http://xerces.apache.org is Apache's XML parser, implements the SAX (recommended), SAX2 and DOM apis in C++, Java and Perl. Released under Apache license.
  • http://xmlsoft.org is Gnome's XML parser, implements SAX/SAX2/DOM, written in C. Released under MIT License.
  • http://expat.sourceforge.net implements the SAX api in C. Released under Apache license.

Other XML projects

Test Files

See NeXML Test Files for some NeXML files to test with.

Outstanding issues

These are not yet in the RFC.

Validation requests

  • Check for tree topological integrity
  • Check for character state referential integrity
  • See NeXML Test Files for files to test with.

General design issues

To meet the requirements for analysis context description, the nexml standard needs to accommodate more metadata about the contents of an entire nexml file:

  • Contents processing: conceivably, a nexml file shouldn't just log how it has changed, but also how it should be processed - i.e. it might specify the steps involved in a workflow, settings used, etc.

Higher-level structure

To meet the requirements for data integration, a higher-level "project" structure that can include multiple nexml files may be necessary. A facility called XInclude exists that allows for merging of xml files into a common Infoset. Unfortunately, this may create id clashes between included files. Some form of processing before inclusion may be necessary, e.g. as described here.

Element changes

Characters

Some data type subclasses need a way to indicate reading frames.

In addition, the characters element needs to be able to provide alternative alignments, possibly using a restricted dictionary that defines, per matrix row, the mapping between location in the unaligned sequence and homologized position, probably using an integervector value type. An attractive suggestion (from MTH) is to attach a vector of integers where the value of each integer is an index in the unaligned sequence, and the number of occurrences of that value is the number of gaps to insert. For example, 0 0 0 means that three gap symbols are prepended to the sequence.

Lastly, characters from different data types might be analyzed jointly. This might be implemented using the class element.

Distances

Nexml needs some notion of pairwise distances (Concept Glossary#Distance_Matrix) between OTUs.

Splits

Nexml needs some notion of OTU bipartitions (Concept Glossary#Bipartition) / splits.

Trees

Compact representations of trees (e.g. compressing of sets of trees by reusing subtrees, representing trees as ancestor functions in integer vectors) may be implemented.


Substitution models

The nexml standard needs to be able to express substitution models, and which characters/trees they apply to. Given that the work to date on this project has been implemented using IDL, there's some chance that this might be transformed directly into XML schema, then added to nexml. During a work meeting in Lawrence, KS, some contributors sketched out an example of what a model description in XML might look like.

Meeting notes