Difference between revisions of "Database Interop Hackathon/Implementations"

From Evolutionary Informatics Working Group
Jump to: navigation, search
(OBO phenotype)
(Specimens within collections)
Line 332: Line 332:
=== the next case ===
=== the next case ===
=== Specimens within collections ===
=== segments of character data in a composite OTU ===
=== segments of character data in a composite OTU ===

Revision as of 13:52, 22 February 2009


This page discusses potential target implementations that demonstrate the goals for the Evolutionary Database Interop Hackathon, to be held at NESCent in March 2009. The overall goal is to expose evolutionary data resource on the web in a machine readable architecture so that they can be integrated in complex work flows and mash-ups. To this end, we suggest their implementing of the stack produced by the evolutionary informatics working group:

  • Syntax - the NeXML data exchange standard
  • Interface/transport - the PhyloWS API for web services
  • Semantics - the CDAO character data analysis ontology

By combining these three components into a single, integrated stack, online data resources will produce output that is easy to parse and validate, whose semantics are well-defined, and whose interface is uniform across data resources.

Current implementations

Some of the online data resources the hackathon seeks to invite for an implementation drive are listed in the table below. The semantics of the output produced by these resources is diverse, including species trees, taxonomies, gene trees, character state matrices and alignments. Also, the programming languages used to implement these services are diverse, including Java, PHP, Python and Perl. This situation goes a long way to explain why standard industry practices (write a WSDL, generate client and server bindings, implement service) have not seen wide adoption in the evolutionary informatics community: different resources have, semantically and syntactically, different inputs and outputs, and are implemented in languages whose support for web services (especially WS-*) is sometimes incomplete.

ResourceExportable objectsImplementation language
GMOD ConceptGlossary#Natural Variation, ConceptGlossary#Polymorphism, Syntenic regions Java, Perl, C, ActionScript
Tree of Life ConceptGlossary#Species_Tree Java
pPOD ?? Java
PhyTome ConceptGlossary#Family_alignment,ConceptGlossary#Phylogenetic_Tree PHP
uBio ConceptGlossary#Taxon, ConceptGlossary#Taxonomic_Rank, ConceptGlossary#Organismal_Taxonomy PHP
TimeTree ConceptGlossary#Species_Tree PHP
PhyloFacts ConceptGlossary#Transition_Model, ConceptGlossary#Phylogenetic_Tree, ConceptGlossary#Family_alignment PHP
MorphBank ConceptGlossary#Character-State_Data_Matrix PHP, Java, JavaScript
MorphoBank ConceptGlossary#Character-State_Data_Matrix PHP, Flash/ActionScript
PhylomeDB ConceptGlossary#Phylogenetic_Tree, ConceptGlossary#Family_alignment JavaScript, Python
PhyloTA ConceptGlossary#Phylogenetic_Tree, ConceptGlossary#Family_alignment Perl
TreeFam ConceptGlossary#Phylogenetic_Tree, ConceptGlossary#Family_alignment Perl
Pandit ConceptGlossary#Phylogenetic_Tree, ConceptGlossary#Family_alignment Perl
MicrobesOnline ConceptGlossary#Phylogenetic_Tree, ConceptGlossary#Family_alignment CGI (=Perl?)
HoVerGen ConceptGlossary#Phylogenetic_Tree, ConceptGlossary#Family_alignment CGI (=Perl?), PHP
PaleoDB ConceptGlossary#Taxon, ConceptGlossary#Taxonomic_Rank, ConceptGlossary#Organismal_Taxonomy CGI (=Perl)

Suggested hackathon deliverables

Data retrieval, based on an identifier

This simple reference service returns phylogenetic data that is identifiable by some GUID, such as a ToLWeb accession number. The service is implemented following the PhyloWS/REST proposal, has a CDAO annotated service description and emits NeXML.

Describing the interface

The first step is to formally describe the interface. In general terms, PhyloWS/REST proposes that data retrieval services are exposed using a URL API like this: /phylows/${dataType}/${nameSpace}:${identifier}, where ${dataType} is something like "Tree", "Matrix", etc. ${nameSpace} is a naming authority such as ToLWeb, and ${identifier} is unique within ${nameSpace} (and consequently globally unique). This implies URLs such as /phylows/Tree/ToLWeb:16299, which, when accessed using the GET HTTP method returns a representation of tree 16299.

A standard way to describe this behaviour is to express it in WSDL2.0 - there's a nifty example of wsdl generation and annotation here. At the time of writing, the best free editor for wsdl files comes with the WTP extension for eclipse. The end result is a file such as this one


Graphical representation of this service description.

Implementing the service

A service for the interface described in the previous section can be implemented as an MVC-like application. The controller part of the service needs to find out what the requested Tree ID is. Depending on the implementation language (and whether some advanced web application programming framework is used) the Tree ID is either part of the ${PATH_INFO} environment variable, or encapsulated in some kind of request object. However the Tree ID is retrieved from the request, the next step is to look up the record that the ID refers to. Typically this would be done in a database query. The goal here is to collect all information needed to populate a model object (in this case a tree) that can be serialized to the right return format. Assuming that the return format is NeXML, libraries for perl, python, java and c++ are available that supply model objects.

Once populated, the controller object creates a view using the model objects. In the simplest case, for web applications, this boils down to printing out the XML string representations of the model objects, preceded by the correct response code, e.g. 200 OK, and mime-type, e.g. application/xml. In more complex web application architectures, the string representations of the model objects may be passed to a response object (which in turn is serialized and returned to the client), or the objects may be passed into a template (jsp, Template Toolkit, php) where they are stringified.

Outstanding issues

  • Dearth of support for PHP
  • How to deal with errors (e.g. response codes)
  • Query interface
  • CDAO integration

Metadata support

NeXML has ways of representing character data and trees, whose semantics are implicitly tied to CDAO (through references to the ontology term using SAWSDL syntax). But, many use cases exist where users want to attach other information to these objects. NeXML has a free-form facility to allow this, however, mass adoption of this feature would lead to a soup of annotations lacking clearly defined semantics. Hence, we need to define how ontology-mediated annotations ('metadata' in our definition) can be used. During discussions at NESCent at the pre-meeting we fleshed out some examples of how this would work.

Attaching a concept to an element

In addition to having the NeXML schema (which defined xml schema classes) define the semantics by reference to their CDAO classes, use cases exist where we would need to tie a NeXML instance (in a document) to a CDAO instance. The solution we came with is to enclose RDF inside an "any" value of a dictionary attachment. The RDF specifies the CDAO class and creates a uniquely identifiable instance on the fly.

Assigning an XML element to a type means creating an instance of that type.

<xml> <tree>

 <node id="foo">
   <dict xmlns:cdao="http://evolutionaryontology.org/cdao">
     <any id="bar">
       <cdao:Node rdf:id="baz"/>

<tree> </xml>

Attaching a taxon identifier to an OTU through a relation

An example of why you would want to a NeXML instance in a document to a CDAO instance of a concept is shown below. It extends the previous example to satisfy a common use case: specifying a taxon identifier from some external resource for an otu element. Notice how this uses the CDAO concept "cdao:has_Taxonomy_Reference" to link the otu element (whose ephemeral id is foo) first to a cdao:TU instance (with id baz), which is subsequently linked to an entry in the Teleost taxonomy (whose record id is 1030219).


 <otu id="foo">
   <dict xmlns:cdao="http://evolutionaryontology.org/cdao">
     <any id="bar">
       <cdao:TU rdf:id="baz">


Attaching a concept to an element through a relation

Another use case along similar syntactical lines as the previous example would be to tie a node in a tree to an inferred gene function from the Gene Ontology. Here we use the CDAO construct "has_function" to specify the semantics of the reference to an external resource.

<xml> <tree>

 <node id="foo">
   <dict xmlns:cdao="http://evolutionaryontology.org/cdao">
     <any id="bar">
       <cdao:Node rdf:id="baz">
         <cdao:has_function rdf:resource="http://purl.org/OBO/GO:034"/>

<tree> </xml>

Specimens within collections

Another common use case for external references is one where a NeXML otu element is to be defined as a specimen in a museum, i.e. we want to specify an identifiable collection, and the number of the specimen within it. In this case we suggest using the TDWG Darwin Core syntax, which has constructs for institutions and catalog numbers. A query of the TDWG ontological activities turned up the TDWG core ontology, however, we are unclear about the status and direction of the CoreOntology (and how to use it), so we're leaving that out for now, choosing the mix the DarwinCore syntax with semantics.

<xml> <otu id="foo">

 <dict xmlns:dwc="http://rs.tdwg.org/dwc/dwcore">
   <any id="watever">
     <cdao:TU rdf:id="baz">
       <cdao:has_Specimen_Reference rdf:parseType="rdf:Literal">
         <dwc:InstitutionCode rdf:datatype="xsd:uri">http://purl.org/obo/COLLECTION:0000403</dwc:InstitutionCode>
         <dwc:CatalogNumber rdf:datatype="xsd:string">207388</dwc:CatalogNumber>

</otu> </xml>

Literature References

Use cases exist where the user would want to attach literature citation records to a phylogenetic object. For example if the users wants to track the provenance of data in a meta-analysis. The subsequent syntax of the record itself could simply be the widely used Dublic Core standard. Yes, this does mean mixing syntax and semantics to some extent, but we concluded that it's a reasonable solution because DC at least implies some semantics (albeit overloaded in some cases, regrettably), and it's syntactically concise.

The Dublin Core [guidelines for dc citations] recommend to provide authors (creators), title and publisher, along with a string giving bibliographic citation. However, the Dublin core does not define what a reference is. Therefore, minimally, we need a term that describes the concept of a reference.

Example 1: associate a reference with a tree (or other) element

In this example, we are just associating a reference with a tree, by placing it in the tree element. In XML using nexml conventions, this is what a literature reference would look like:

<xml> <tree id="foo">

 <dict xmlns:dc="http://purl.org/dc/elements/1.1" xmlns:dcterms="http://purl.org/dc/terms">
   <any id="foo235">
     <cdao:Tree rdf:id="bar">
      <cdao:has_Reference rdf:parseType="rdf:resource">
       <dc:creator>Hill, R. V.</dc:creator>
       <dc:title>Integration of Morphological Data Sets for Phylogenetic Analysis of Amniota:
                         The Importance of Integumentary Characters and Increased Taxonomic Sampling</dc:title>
       <dcterms:bibliographicCitation>Systematic Biology 54(4):530-547, 2005</dcterms:bibliographicCitation>

</tree> </xml>

Example 2: associate a reference with a record

<xml> <nexml> <dict xmlns:obi="http://purl.obofoundry.org/obo/obi.owl" xmlns:cdao="http://evolutionaryontology.org/cdao/cdao.owl">

 <any id="foo235">
   <obi:IAO_0000100 rdf:id="foo234">
     <cdao:has_Reference rdf:parseType="Resource">
       <dc:creator>Hill, R. V.</dc:creator>
       <dc:title>Integration of Morphological Data Sets for Phylogenetic Analysis of Amniota:
                     The Importance of Integumentary Characters and Increased Taxonomic Sampling</dc:title>
       <dcterms:bibliographicCitation>Systematic Biology 54(4):530-547, 2005</dcterms:bibliographicCitation>

</dict> </nexml> </xml>

Example 3: specifying that a reference represents "supporting evidence" for a nexml element

Work in progress

OBO phenotype

<xml> <state id="x88913" label="present" symbol="1">

 <dict xmlns:cdao="http://www.evolutionaryontology.org/cdao.owl"
   <any id="foo345">
     <cdao:Categorical rdf:id="foo456">
       <cdao:has_ExternalReference rdf:parseType="Literal">
             <phen:typeref about="TAO:0000203"/>
             <phen:typeref about="PATO:0000467"/>

</state> </xml>

Segments of character data

the next case

segments of character data in a composite OTU