Ontology Alignment Evaluation Initiative - OAEI-2012 Campaign

Results OAEI 2012::Large BioMed Track

Results OAEI 2012 FMA-NCI matching problem

FMA-NCI small fragments

This year we obtained very high level participation and 11 systems/configurations obtained, on average, an F-measure over 0.80 for the matching problem involving the small fragments of FMA and NCI. GOMMA-bk obtained the best results in terms of both recall and F-measure while ServOMap provided the most precise alignments. LogMap and LogMap-noe provided the same results since the input ontologies are already small fragments of FMA and NCI and thus, the overlapping estimation performed by LogMap did not have any impact. In general, as expected, precision increases when comparing against the original UMLS mapping set, while recall decreases.

Our baseline provided very good results in terms of F-measure and outperformed 8 of the participating systems. MaasMatch and Hertuda provided competitive results in terms of recall, but the low precision damaged the final F-measure. MapSSS and AUTOMSv2 provided a set of mappings with high precision, however, the F-measure was damaged due to the low recall of their mappings.

The runtimes were very positive in general and 8 systems completed the task in less than 2 minutes. MapSSS required less than 10 minutes, while Hertuda and HotMatch required around 1 hour. Finally, MaasMatch, AUTOMSv2 and Wmatch needed 8, 17 and 18 hours to complete the task, respectively.

Regarding mapping coherence, only LogMap (with its two variants) generates an almost clean output. In the table, we can appreciate that even the most precise mappings (ServOMap or YAM++) lead to a huge amount of unsatisfiable classes when reasoning together with the input ontologies; and thus, it proves the importance of using techniques to assess the coherence of the generated alignments. Unfortunately, LogMap and CODI are the unique systems (participating in the OAEI 2012) that have shown to use such techniques.

FMA-NCI big fragments

AUTOMSv2, HotMatch, Hertuda, Wmatch and MaasMatch failed to complete the task involving the big fragments of FMA and NCI after more than 24 hours of execution. Runtimes were in line with the small matching task, apart from the ones for MapSSS and AROMA which suffered an important increase.

YAM++ provided the best results in terms of F-measure, whereas GOMMA-bk and ServOMap got the best recall and precision, respectively. F-measures have decreased considerably with respect to the small matching task. This is mostly due to the fact that this matching task involves more possible candidate mappings than the previous one. Nevertheless, seven systems outperformed our baseline and provided high quality mapping sets in terms of both precision and recall. Only, MapSSS and AROMA provided worse results in terms of both precision and recall than LogMapLt.

Regarding mapping coherence, as in the previous task, only LogMap (with its two variants) generates an almost clean output where the mappings together with the input ontologies only lead to 5 unsatisfiable classes.

FMA-NCI whole ontologies

AROMA and MapSSS failed to complete the matching task involving the whole FMA and NCI ontologies in less than 24 hours.

As in the previous task, the remaining 7 matching systems generated high quality mapping sets. YAM++ provided the best results in terms of F-measure, whereas GOMMA-bk and ServOMap got the best recall and precision, respectively. LogMap with its two configurations provided an almost clean output and only 9 classes where unsatisfiable after reasoning with the input ontologies and the computed mappings.

Runtimes were also very positive. YAM++ was slightly slower than the other systems, which gave the outputs in less than 5 minutes, and required around 20 minutes to complete the task.