View source for Database Interop Hackathon/Implementations

= Summary =
This page discusses potential target implementations that demonstrate the goals for the Evolutionary [[Database Interop Hackathon]], to be held at NESCent in March 2009. The overall goal is to expose evolutionary data resource on the web in a machine readable architecture so that they can be integrated in complex work flows and mash-ups. To this end, we suggest their implementing of the stack produced by the evolutionary informatics working group:
* ''Syntax'' - the [[Future_Data_Exchange_Standard|NeXML]] data exchange standard
* ''Interface/transport'' - the [[PhyloWS]] API for web services
* ''Semantics'' - the [[CDAO]] character data analysis ontology
By combining these three components into a single, integrated stack, online data resources will produce output that is easy to parse and validate, whose semantics are well-defined, and whose interface is uniform across data resources.

= Current implementations =
Some of the online data resources the hackathon seeks to invite for an implementation drive are listed in the table below. The semantics of the output produced by these resources is diverse, including species trees, taxonomies, gene trees, character state matrices and alignments. Also, the programming languages used to implement these services are diverse, including Java, PHP, Python and Perl. This situation goes a long way to explain why standard industry practices (write a WSDL, generate client and server bindings, implement service) have not seen wide adoption in the evolutionary informatics community: different resources have, semantically and syntactically, different inputs and outputs, and are implemented in languages whose support for web services (especially WS-*) is sometimes incomplete.
<table>
<tr><th align="left" style="border-bottom:1px solid silver">Resource</th><th align="left" style="border-bottom:1px solid silver">Exportable objects</th><th align="left" style="border-bottom:1px solid silver">Implementation language</th></tr>

<tr>
<td>[http://gmod.org/ GMOD]</td>
<td>[[ConceptGlossary#Natural Variation]], [[ConceptGlossary#Polymorphism]], Syntenic regions</td>
<td>Java, Perl, C, ActionScript</td>
</tr>

<tr>
<td>[http://tolweb.org/ Tree of Life]</td>
<td>[[ConceptGlossary#Species_Tree]]</td>
<td>Java</td>
</tr>

<tr>
<td>[http://phylodata.seas.upenn.edu pPOD]</td>
<td>??</td>
<td>Java</td>
</tr>

<tr>
<td>[http://www.phytome.org PhyTome]</td>
<td>[[ConceptGlossary#Family_alignment]],[[ConceptGlossary#Phylogenetic_Tree]]</td>
<td>PHP</td>
</tr>

<tr>
<td>[http://www.ubio.org uBio]</td>
<td>[[ConceptGlossary#Taxon]], [[ConceptGlossary#Taxonomic_Rank]], [[ConceptGlossary#Organismal_Taxonomy]]</td>
<td>PHP</td>
</tr>

<tr>
<td>[http://timetree.org/index.php TimeTree]</td>
<td>[[ConceptGlossary#Species_Tree]]</td>
<td>PHP</td>
</tr>

<tr>
<td>[http://phylogenomics.berkeley.edu PhyloFacts]</td>
<td>[[ConceptGlossary#Transition_Model]], [[ConceptGlossary#Phylogenetic_Tree]], [[ConceptGlossary#Family_alignment]]</td>
<td>PHP</td>
</tr>

<tr>
<td>[http://www.morphbank.net MorphBank]</td>
<td>[[ConceptGlossary#Character-State_Data_Matrix]]</td>
<td>PHP, Java, JavaScript</td>
</tr>

<tr>
<td>[http://morphobank.geongrid.org MorphoBank]</td>
<td>[[ConceptGlossary#Character-State_Data_Matrix]]</td>
<td>PHP, Flash/ActionScript</td>
</tr>

<tr>
<td>[http://phylomedb.bioinfo.cipf.es/index.html PhylomeDB]</td>
<td>[[ConceptGlossary#Phylogenetic_Tree]], [[ConceptGlossary#Family_alignment]]</td>
<td>JavaScript, Python</td>
</tr>

<tr>
<td>[http://loco.biosci.arizona.edu/pb/ PhyloTA]</td>
<td>[[ConceptGlossary#Phylogenetic_Tree]], [[ConceptGlossary#Family_alignment]]</td>
<td>Perl</td>
</tr>

<tr>
<td>[http://www.treefam.org/ TreeFam]</td>
<td>[[ConceptGlossary#Phylogenetic_Tree]], [[ConceptGlossary#Family_alignment]]</td>
<td>Perl</td>
</tr>

<tr>
<td>[http://www.ebi.ac.uk/goldman-srv/pandit/ Pandit]</td>
<td>[[ConceptGlossary#Phylogenetic_Tree]], [[ConceptGlossary#Family_alignment]]</td>
<td>Perl</td>
</tr>

<tr>
<td>[http://www.microbesonline.com MicrobesOnline]</td>
<td>[[ConceptGlossary#Phylogenetic_Tree]], [[ConceptGlossary#Family_alignment]]</td>
<td>CGI (=Perl?)</td>
</tr>

<tr>
<td>[http://pbil.univ-lyon1.fr/databases/hovergen.php HoVerGen]</td>
<td>[[ConceptGlossary#Phylogenetic_Tree]], [[ConceptGlossary#Family_alignment]]</td>
<td>CGI (=Perl?), PHP</td>
</tr>

<tr>
<td>[[PaleoDB]]</td>
<td>[[ConceptGlossary#Taxon]], [[ConceptGlossary#Taxonomic_Rank]], [[ConceptGlossary#Organismal_Taxonomy]]</td>
<td>CGI (=Perl)</td>
</tr>

</table>

= Suggested hackathon deliverables =
== Data retrieval, based on an identifier ==
This simple reference service returns phylogenetic data that is identifiable by some GUID, such as a [http://tolweb.org ToLWeb] accession number. The service is implemented following the [[PhyloWS/REST]] proposal, has a [[CDAO]] annotated service description and emits [[Future_Data_Exchange_Standard|NeXML]].

=== Describing the interface ===
The first step is to formally describe the interface. In general terms, PhyloWS/REST proposes that data retrieval services are exposed using a URL API like this: '''<code>/phylows/${dataType}/${nameSpace}:${identifier}</code>''', where '''<code>${dataType}</code>''' is something like "Tree", "Matrix", etc. '''<code>${nameSpace}</code>''' is a naming authority such as ToLWeb, and '''<code>${identifier}</code>''' is unique within '''<code>${nameSpace}</code>''' (and consequently globally unique). This implies URLs such as '''<code>/phylows/Tree/ToLWeb:16299</code>''', which, when accessed using the [http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods GET HTTP method] returns a representation of tree [http://tolweb.org/16299 16299].

A standard way to describe this behaviour is to express it in [http://en.wikipedia.org/wiki/Web_Services_Description_Language WSDL2.0] - there's a nifty example of wsdl generation and annotation [http://www.keith-chapman.org/2008/09/restfull-mashup-with-wsdl-20-wso2.html here]. At the time of writing, the best free editor for wsdl files comes with the WTP extension for [http://eclipse.org eclipse]. The end result is a file such as [http://nexml.org/nexml/phylows/tolweb?wsdl this one]
<div style="1px solid silver">
[[Image:Tolrest.jpg]]

''Graphical representation of [http://nexml.org/nexml/phylows/tolweb?wsdl this] service description.''
</div>

=== Implementing the service ===
A service for the interface described in the previous section can be implemented as an MVC-like application. The '''controller''' part of the service needs to find out what the requested Tree ID is. Depending on the implementation language (and whether some advanced web application programming framework is used) the Tree ID is either part of the '''<code>${PATH_INFO}</code>''' environment variable, or encapsulated in some kind of request object. However the Tree ID is retrieved from the request, the next step is to look up the record that the ID refers to. Typically this would be done in a database query. The goal here is to collect all information needed to populate a '''model''' object (in this case a tree) that can be serialized to the right return format. Assuming that the return format is NeXML, libraries for perl, python, java and c++ are available that supply model objects.

Once populated, the controller object creates a view using the model objects. In the simplest case, for web applications, this boils down to printing out the XML string representations of the model objects, preceded by the correct [http://en.wikipedia.org/wiki/List_of_HTTP_status_codes response code], e.g. <code>200 OK</code>, and [http://en.wikipedia.org/wiki/MIME mime-type], e.g. <code>application/xml</code>. In more complex web application architectures, the string representations of the model objects may be passed to a response object (which in turn is serialized and returned to the client), or the objects may be passed into a template (jsp, Template Toolkit, php) where they are stringified.
=== Outstanding issues ===
* Dearth of support for PHP
* How to deal with errors (e.g. response codes)
* Query interface
* CDAO integration

== Metadata support ==

NeXML has ways of representing character data and trees, whose semantics are implicitly tied to CDAO (through references to the ontology term using SAWSDL syntax). But, many use cases exist where users want to attach other information to these objects. NeXML has a free-form facility to allow this, however, mass adoption of this feature would lead to a soup of annotations lacking clearly defined semantics. Hence, we need to define how ontology-mediated annotations ('metadata' in our definition) can be used. During discussions at NESCent at the pre-meeting we fleshed out some examples of how this would work.

=== Literature References ===

Dublin core namespace has title, author, etc., but does not have the concept of a "reference".

This version is semantically explicit and syntactically concise.  It assigns an RDF term label (making it human readable) and an RDF term identifier (about) to the dict element (making it globally unique) from CDAO or some other ontology; and the content of the dict element comes from a namespace that also is clearly identified.

<xml>
  <dict rdf:label="CDAO Reference" rdf:about="http://purl.org/CDAO/CDAO:723" xsi:type="nex:DC">
    <any id="whatever">
      <dc:author>Larry</dc:author>
      <dc:title>About Larry</dc:title>
    </any>
    <any id="another one">
      <dc:author>Mary</dc:author>
      <dc:title>About Mary</dc:title>
      . . .
    </any>
  </dict>
  <dict rdf:label="supporting evidence" rdf:about="http://purl.org/CDAO/CDAO:334" xsi:type="nex:DC">
    <valueref any="whatever"/>
    <valueref any="another one"/>
  </dict>
</xml>

=== Attaching a concept to an element ===

Assigning an XML element to a type means creating an instance of that type.

<xml>
<tree>
  <node id="9887438">
    <dict xmlns:cdao="http://evolutionaryontology.org/cdao">
      <any id="wotever">
        <cdao:Node rdf:id="#CDAO:658"/>
      </any>
    </dict>
  </node>
<tree>
</xml>

=== Attaching a taxon identifier to an OTU through a relation ===

<xml>
  <otu id="9887438">
    <dict xmlns:cdao="http://evolutionaryontology.org/cdao">
      <any id="watever">
        <cdao:TU rdf:id="#CDAO:658">
          <!-- here is where we identify the relation and assign a value -->
          <!-- which is here a concept from another ontology -->
          <cdao:has_External_Reference
              rdf:resource="http://purl.org/OBO/TTO:1030219"/>
        </cdao:TU>
      </any>
    </dict>
  </otu>
</xml>

=== Attaching a concept to an element through a relation ===

<xml>
<tree>
  <node id="9887438">
    <dict xmlns:cdao="http://evolutionaryontology.org/cdao">
      <any id="watever">
        <cdao:Node rdf:id="#CDAO:658">
          <!-- here is where we identify the relation and assign a value -->
          <!-- which is here a concept from another ontology -->
          <cdao:has_function rdf:resource="http://purl.org/OBO/GO:034"/>
        </cdao:Node>
      </any>
    </dict>
  </node>
<tree>
</xml>

=== Specimens within collections ===

<xml>
<otu>
  <dict xmlns:dwc="http://rs.tdwg.org/dwc/dwcore">
    <any id="watever">
      <rdf:Object rdf:id="specimen01">
         <dwc:InstitutionCode rdf:resource="http://purl.org/obo/COLLECTION:0000403"/>
         <dwc:CatalogNumber rdf:datatype="xsd:string">207388</dwc:CatalogNumber>
      </rdf:Object>
      <cdao:TU rdf:id="#CDAO:658">
        <cdao:has_External_Reference rdf:resource="#specimen01">
      </cdao:TU>
    </any>
  </dict
</otu>
</xml>

Better yet:

<xml>
<otu>
  <dict xmlns:dwc="http://rs.tdwg.org/dwc/dwcore">
    <any id="watever">
      <rdf:Object rdf:id="specimen01">
         <dwc:InstitutionCode rdf:resource="http://purl.org/obo/COLLECTION:0000403"/>
         <dwc:CatalogNumber rdf:datatype="xsd:string">207388</dwc:CatalogNumber>
      </rdf:Object>
      <cdao:TU rdf:id="#CDAO:658">
        <cdao:has_Specimen_Reference rdf:resource="#specimen01">
      </cdao:TU>
    </any>
  </dict
</otu>
</xml>

Or, using the [http://wiki.tdwg.org/twiki/bin/view/TAG/CoreOntology TDWG core ontology]:

: ''Not clear. Let's ask the TDWG people.''

=== OBO  phenotype ===


=== Segments of character data ===

=== the next case ===

=== Specimens within collections ===
=== segments of character data in a composite OTU ===

[[Category:DB Interop Hackathon]]
[[Category:Interoperability]]
[[Category:Nexml]]
[[Category:CDAO]]