Results OAEI 2018::Large BioMed Track

Contact

If you have any question/suggestion related to the results of this track or if you notice any kind of error (wrong numbers, incorrect information on a matching system, etc.), feel free to write an email to ernesto [.] jimenez [.] ruiz [at] gmail [.] com

Evaluation setting

We have run the evaluation in a Ubuntu 18 Laptop with an Intel Core i9-8950HK CPU @ 2.90GHz x 12 and allocating 25Gb of RAM.

Precision, Recall and F-measure have been computed with respect to a UMLS-based reference alignment. Systems have been ordered in terms of F-measure.

Unique mappings show the number of mappings computed by a system that are not predicted by any of the other participants (including variants).

Check out the supporting scripts to reproduce the evaluation: https://github.com/ernestojimenezruiz/oaei-evaluation

Participation and success

In the OAEI 2018 largebio track 10 out of 18 participating systems have been able to complete at least one of the tasks of the largebio track within a 6 hours timeout. Seven systems were able to complete all six largebio tracks.

Use of background knowledge

LogMapBio uses BioPortal as mediating ontology provider, that is, it retrieves from BioPortal the most suitable top-10 ontologies for the matching task.

LogMap uses normalisations and spelling variants from the general (biomedical) purpose SPECIALIST Lexicon.

AML has three sources of background knowledge which can be used as mediators between the input ontologies: the Uber Anatomy Ontology (Uberon), the Human Disease Ontology (DOID) and the Medical Subject Headings (MeSH).

XMap and Lily rely on a subset of synonyms provided by the UMLS Metathesaurus . Since the largebio reference alignment is also based on the UMLS-Metathesaurus we disallowed the use of this UMLS-based dictionary. XMap still produced competitive results, while Lily produced an empty set of alignments.

Alignment coherence

Together with Precision, Recall, F-measure and Runtimes we have also evaluated the coherence of alignments. We have reported (1) number of unsatisfiabilities when reasoning with the input ontologies together with the computed mappings, and (2) the ratio/degree of unsatisfiable classes with respect to the size of the union of the input ontologies.

We have used the OWL 2 reasoner HermiT to compute the number of unsatisfiable classes. For the cases in which HermiT could not cope with the input ontologies and the mappings (in less than 2 hours) we have provided a lower bound on the number of unsatisfiable classes (indicated by ≥) using the OWL 2 EL reasoner ELK.

As in previous OAEI editions, only three systems have shown mapping repair facilities, namely: AML, LogMap (including its LogMapBio variant) and XMap (uses Alcomo repair system). All three systems produce relatively clean outputs in FMA-NCI and FMA-SNOMED cases; however in the SNOMED-NCI cases AML and XMap mappings lead to a number of unsatisfiable classes. XMap seems to deactivate the repair facility for the SNOMED-NCI case. The results also show that even the most precise alignment sets may lead to a large amount of unsatisfiable classes. This proves the importance of using techniques to assess the coherence of the generated alignments.

1. System runtimes and task completion

**Table 1:** System runtimes (s) and task completion.
System	FMA-NCI		FMA-SNOMED		SNOMED-NCI		Average	# Tasks
System	Task 1	Task 2	Task 3	Task 4	Task 5	Task 6	Average	# Tasks
LogMapLt	1	6	1	9	5	11	6	6
DOME	2	12	2	20	10	24	12	6
AML	24	55	68	94	346	168	126	6
XMap	7	65	26	299	124	427	158	6
LogMap	6	51	33	287	123	475	163	6
FCAMapX	40	881	91	1,736	833	2,377	993	6
LogMapBio	701	1,072	890	1,840	1,500	2,942	1,491	6
POMAP++	254	-	779	-	6,312	-	2,448	3
ALOD2Vec	342	-	1,400	-	9,209	-	3,650	3
KEPLER	588	-	4,163	-	-	-	2,376	2
# Systems	10	7	10	7	9	7	1,142	50

2. Results for the FMA-NCI matching problem

Task 1: FMA-NCI small fragments

**Table 2:** Results for the largebio task 1.
System	Time (s)	# Mappings	# Unique	Scores			Incoherence Analysis
System	Time (s)	# Mappings	# Unique	Precision	Recall	F-measure	Unsat.	Degree
AML	24	2,723	77	0.958	0.910	0.933	2	0.020%
FCAMapX	40	2,828	89	0.948	0.911	0.929	4,703	46.2%
LogMapBio	701	2,776	14	0.941	0.902	0.921	2	0.020%
LogMap	6	2,747	3	0.944	0.897	0.920	2	0.020%
ALOD2Vec	342	2,528	6	0.972	0.839	0.901	2,060	20.2%
KEPLER	588	2,506	42	0.960	0.831	0.891	3,707	36.4%
POMAP++	254	2,414	9	0.979	0.814	0.889	993	9.751%
LogMapLt	1	2,480	11	0.967	0.819	0.887	2,104	20.7%
XMap	7	2,315	2	0.977	0.783	0.869	2	0.020%
DOME	2	2,248	1	0.985	0.764	0.861	324	3.181%

Task 2: FMA-NCI whole ontologies

**Table 3:** Results for the largebio task 2.
System	Time (s)	# Mappings	# Unique	Scores			Incoherence Analysis
System	Time (s)	# Mappings	# Unique	Precision	Recall	F-measure	Unsat.	Degree
AML	55	2,968	311	0.838	0.872	0.855	10	0.007%
LogMap	51	2,701	0	0.856	0.808	0.831	9	0.006%
LogMapBio	1,072	2,860	39	0.830	0.831	0.830	9	0.006%
XMap	65	2,415	52	0.878	0.742	0.804	9	0.006%
FCAMapX	881	3,607	443	0.665	0.841	0.743	84,601	58.1%
LogMapLt	6	3,458	250	0.676	0.819	0.741	26,267	18.0%
DOME	12	2,383	10	0.803	0.668	0.729	3,595	2.5%

3. Results for the FMA-SNOMED matching problem

Task 3: FMA-SNOMED small fragments

**Table 4:** Results for the largebio task 3.
System	Time (s)	# Mappings	# Unique	Scores			Incoherence Analysis
System	Time (s)	# Mappings	# Unique	Precision	Recall	F-measure	Unsat.	Degree
FCAMapX	91	7,582	574	0.955	0.815	0.879	13,680	58.0%
AML	68	6,988	415	0.923	0.762	0.835	0	0.000%
LogMapBio	890	6,319	8	0.947	0.693	0.800	1	0.004%
LogMap	33	6,282	1	0.947	0.690	0.798	1	0.004%
XMap	26	5,815	36	0.962	0.647	0.774	0	0.000%
KEPLER	4,163	4,005	375	0.822	0.424	0.559	3,335	14.1%
POMAP++	779	2,163	49	0.906	0.260	0.404	894	3.793%
ALOD2Vec	1,400	1,727	47	0.941	0.213	0.347	818	3.471%
LogMapLt	1	1,642	2	0.968	0.208	0.342	771	3.271%
DOME	2	1,530	0	0.988	0.198	0.330	751	3.186%

Task 4: FMA whole ontology with SNOMED large fragment

**Table 5:** Results for the largebio task 4.
System	Time (s)	# Mappings	# Unique	Scores			Incoherence Analysis
System	Time (s)	# Mappings	# Unique	Precision	Recall	F-measure	Unsat.	Degree
FCAMapX	1,736	7,971	1,258	0.819	0.762	0.789	76,120	37.8%
AML	94	6,571	462	0.882	0.687	0.772	0	0%
LogMapBio	1,840	6,471	31	0.834	0.650	0.731	0	0%
LogMap	287	6,393	0	0.840	0.645	0.730	0	0%
XMap	299	6,749	1,217	0.723	0.608	0.661	0	0%
LogMapLt	9	1,820	56	0.851	0.208	0.334	4,389	2.2%
DOME	20	1,588	1	0.941	0.197	0.326	4,314	2.1%

4. Results for the SNOMED-NCI matching problem

Task 5: SNOMED-NCI small fragments

**Table 6:** Results for the largebio task 5.
System	Time (s)	# Mappings	# Unique	Scores			Incoherence Analysis
System	Time (s)	# Mappings	# Unique	Precision	Recall	F-measure	Unsat.	Degree
AML	346	14,435	1,699	0.878	0.737	0.801	≥3,244	≥4.3%
FCAMapX	833	13,789	704	0.878	0.703	0.781	≥62,854	≥83.7%
LogMapBio	1,500	12,678	139	0.912	0.672	0.774	≥0	≥0%
LogMap	123	12,414	0	0.922	0.665	0.773	≥0	≥0%
LogMapLt	5	10,921	65	0.893	0.566	0.693	≥60,447	≥80.5%
XMap	124	12,125	1,114	0.835	0.588	0.690	≥49,878	≥66.4%
POMAP++	6,312	10,895	356	0.889	0.563	0.689	≥57,713	≥76.9%
DOME	10	9,321	8	0.922	0.499	0.648	≥44,232	≥58.9%
ALOD2Vec	9,209	12,882	2,180	0.743	0.556	0.636	≥49,480	≥65.9%

Task 6: NCI whole ontology with SNOMED large fragment

**Table 7:** Results for the largebio task 6.
System	Time (s)	# Mappings	# Unique	Scores			Incoherence Analysis
System	Time (s)	# Mappings	# Unique	Precision	Recall	F-measure	Unsat.	Degree
AML	168	13,176	1,230	0.904	0.668	0.768	≥720	≥0.4%
FCAMapX	2,377	15,383	1,670	0.796	0.680	0.733	≥151,001	≥79.8%
LogMapBio	2,942	13,098	231	0.854	0.627	0.723	≥6	≥0.003%
LogMap	475	12,276	0	0.867	0.596	0.706	≥4	≥0.002%
LogMapLt	11	12,864	720	0.798	0.566	0.662	≥150,454	≥79.5%
DOME	24	9,702	42	0.907	0.485	0.632	≥101,436	≥53.6%
XMap	427	16,271	4,432	0.640	0.582	0.610	≥157,345	≥83.2%