Ontology Alignment Evaluation Initiative - OAEI-2016 Campaign

Large BioMed Track

Results OAEI 2016::Large BioMed Track

Contact

If you have any question/suggestion related to the results of this track or if you notice any kind of error (wrong numbers, incorrect information on a matching system, etc.), feel free to write an email to ernesto [at] cs [.] ox [.] ac [.] uk or ernesto [.] jimenez [.] ruiz [at] gmail [.] com

Evaluation setting

We have run the evaluation in a Ubuntu Laptop with an Intel Core i7-4600U CPU @ 2.10GHz x 4 and allocating 15Gb of RAM.

Precision, Recall and F-measure have been computed with respect to a UMLS-based reference alignment. Systems have been ordered in terms of F-measure.

Participation and success

In the OAEI 2016 largebio track 11 out of 21 participating OAEI 2016 systems have been able to cope with at least one of the tasks of the largebio track with a 2 hours timeout (we plan to extend this timeout after the workshop).

Use of background knowledge

LogMapBio uses BioPortal as mediating ontology provider, that is, it retrieves from BioPortal the most suitable top-10 ontologies for the matching task.

LogMap uses normalisations and spelling variants from the general (biomedical) purpose UMLS Lexicon.

AML has three sources of background knowledge which can be used as mediators between the input ontologies: the Uber Anatomy Ontology (Uberon), the Human Disease Ontology (DOID) and the Medical Subject Headings (MeSH).

XMAP uses synonyms provided by the UMLS Metathesaurus. Note that matching systems using UMLS-Metathesaurus as background knowledge will have a notable advantage since the largebio reference alignment is also based on the UMLS-Metathesaurus.

Alignment coherence

Together with Precision, Recall, F-measure and Runtimes we have also evaluated the coherence of alignments. We have reported (1) number of unsatisfiabilities when reasoning with the input ontologies together with the computed mappings, and (2) the ratio/degree of unsatisfiable classes with respect to the size of the union of the input ontologies.

We have used the OWL 2 reasoner HermiT to compute the number of unsatisfiable classes. For the cases in which HermiT could not cope with the input ontologies and the mappings (in less than 2 hours) we have provided a lower bound on the number of unsatisfiable classes (indicated by ≥) using the OWL 2 EL reasoner ELK.

In this OAEI edition, only three systems have shown mapping repair facilities, namely: AML, LogMap (including LogMapBio variant) and XMap. The results show that even the most precise alignment sets may lead to a huge amount of unsatisfiable classes. This proves the importance of using techniques to assess the coherence of the generated alignments.


1. System runtimes and task completion

System FMA-NCI FMA-SNOMED SNOMED-NCI Average # Tasks
Task 1 Task 2 Task 3 Task 4 Task 5 Task 6
LogMapLite 1 10 2 18 8 18 10 6
AML 35 72 98 166 537 376 214 6
LogMap 10 80 60 433 177 699 243 6
LogMapBio 1,712 1,188 1,180 2,156 3,757 4,322 2,386 6
XMap 17 116 54 366 267 - 164 5
FCA_Map 236 - 1,865 - - - 1,051 2
Lily 699 - - - - - 699 1
LYAM 1,043 - - - - - 1,043 1
DKP-AOM 1,547 - - - - - 1,547 1
DKP-AOM-Lite 1,698 - - - - - 1,698 1
Alin 5,811 - - - - - 5,811 1
# Systems 11 5 6 5 5 4 1,351 36
Table 1: System runtimes (s) and task completion.


2. Results for the FMA-NCI matching problem

Task 1: FMA-NCI small fragments

System Time (s) # Mappings Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
XMap* 17 2,649 0.977 0.901 0.937 2 0.019%
FCA_Map 236 2,834 0.954 0.917 0.935 4,729 46.0%
AML 35 2,691 0.963 0.902 0.931 2 0.019%
LogMap 10 2,747 0.949 0.901 0.924 2 0.019%
LogMapBio 1,712 2,817 0.935 0.910 0.923 2 0.019%
LogMapLite 1 2,483 0.967 0.819 0.887 2,045 19.9%
Average 1,164 2,677 0.852 0.779 0.804 2,434 23.7%
LYAM 1,043 3,534 0.721 0.889 0.796 6,880 66.9%
Lily 699 3,374 0.603 0.721 0.657 9,273 90.2%
Alin 5,811 1,300 0.995 0.455 0.625 0 0.000%
DKP-AOM-Lite 1,698 2,513 0.652 0.577 0.612 1,924 18.7%
DKP-AOM 1,547 2,513 0.652 0.577 0.612 1,924 18.7%
Table 2: Results for the largebio task 1.

Task 2: FMA-NCI whole ontologies

System Time (s) # Mappings Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
XMap* 116 2,681 0.902 0.847 0.874 9 0.006%
AML 72 2,968 0.838 0.872 0.855 10 0.007%
LogMap 80 2,693 0.854 0.802 0.827 9 0.006%
LogMapBio 1,188 2,924 0.818 0.835 0.826 9 0.006%
Average 293 2,948 0.817 0.835 0.824 5,303 3.6%
LogMapLite 10 3,477 0.673 0.820 0.739 26,478 18.1%
Table 3: Results for the largebio task 2.


3. Results for the FMA-SNOMED matching problem

Task 3: FMA-SNOMED small fragments

System Time (s) # Mappings Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
XMap* 54 7,311 0.989 0.846 0.912 0 0.000%
FCA_Map 1,865 7,649 0.936 0.803 0.865 14,603 61.8%
AML 98 6,554 0.953 0.727 0.825 0 0.000%
LogMapBio 1,180 6,357 0.944 0.696 0.801 1 0.004%
LogMap 60 6,282 0.948 0.690 0.799 1 0.004%
Average 543 5,966 0.957 0.662 0.758 2,562 10.8%
LogMapLite 2 1,644 0.968 0.209 0.343 771 3.3%
Table 4: Results for the largebio task 3.

Task 4: FMA whole ontology with SNOMED large fragment

System Time (s) # Mappings Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
XMap* 366 7,361 0.965 0.843 0.900 0 0.000%
AML 166 6,571 0.882 0.687 0.773 0 0.000%
LogMap 433 6,281 0.839 0.634 0.722 0 0.000%
LogMapBio 2,156 6,520 0.808 0.640 0.714 0 0.000%
Average 627 5,711 0.869 0.602 0.689 877 0.4%
LogMapLite 18 1,822 0.852 0.209 0.335 4,389 2.2%
Table 5: Results for the largebio task 4.


4. Results for the SNOMED-NCI matching problem

Task 5: SNOMED-NCI small fragments

System Time (s) # Mappings Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
AML 537 13,584 0.904 0.713 0.797 ≥0 ≥0.00%
LogMap 177 12,371 0.922 0.663 0.771 ≥0 ≥0.00%
LogMapBio 3,757 12,960 0.896 0.675 0.770 ≥0 ≥0.00%
Average 949 13,302 0.905 0.636 0.746 12,090 16.1%
XMap* 267 16,657 0.911 0.564 0.697 ≥0 ≥0.00%
LogMapLite 8 10,942 0.892 0.567 0.693 ≥60,450 ≥80.4%
Table 6: Results for the largebio task 5.


Task 6: NCI whole ontology with SNOMED large fragment

System Time (s) # Mappings Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
AML 376 13,175 0.904 0.668 0.768 ≥2 ≥0.001%
LogMapBio 4,322 13,477 0.842 0.637 0.725 ≥6 ≥0.003%
Average 1,353 12,942 0.853 0.617 0.716 37,667 19.9%
LogMap 699 12,222 0.870 0.596 0.708 ≥4 ≥0.002%
LogMapLite 18 12,894 0.797 0.567 0.663 ≥150,656 ≥79.5%
Table 7: Results for the largebio task 6.

* Uses background knowledge based on the UMLS-Metathesaurus as the largebio reference alignments.



Contact: ernesto [.] jimenez [.] ruiz [at] gmail [.] com
http://www.cs.ox.ac.uk/isg/projects/SEALS/oaei/