Overview of the CLEF eHealth evaluation lab 2015

Lorraine Goeuriot, Liadh Kelly, Hanna Suominen, Leif Hanlen, Aurèlie Nèvèol, Cyril Grouin, João Palotti, Guido Zuccon

Research output: A Conference proceeding or a Chapter in BookConference contributionpeer-review

63 Citations (Scopus)


This paper reports on the 3rd CLEFeHealth evaluation lab, which continues our evaluation resource building activities for the medical domain. In this edition of the lab, we focus on easing patients and nurses in authoring, understanding, and accessing eHealth information. The 2015 CLEFeHealth evaluation lab was structured into two tasks, focusing on evaluating methods for information extraction (IE) and information retrieval (IR). The IE task introduced two new challenges. Task 1a focused on clinical speech recognition of nursing handover notes; Task 1b focused on clinical named entity recognition in languages other than English, specifically French. Task 2 focused on the retrieval of health information to answer queries issued by general consumers seeking information to understand their health symptoms or conditions. The number of teams registering their interest was 47 in Tasks 1 (2 teams in Task 1a and 7 teams in Task 1b) and 53 in Task 2 (12 teams) for a total of 20 unique teams. The best system recognized 4, 984 out of 6, 818 test words correctly and generated 2, 626 incorrect words (i.e., 38.5% error) in Task 1a; had the F-measure of 0.756 for plain entity recognition, 0.711 for normalized entity recognition, and 0.872 for entity normalization in Task 1b; and resulted in P@10 of 0.5394 and nDCG@10 of 0.5086 in Task 2. These results demonstrate the substantial community interest and capabilities of these systems in addressing challenges faced by patients and nurses. As in previous years, the organizers have made data and tools available for future research and development.

Original languageEnglish
Title of host publicationExperimental IR Meets Multilinguality, Multimodality, and Interaction
Subtitle of host publication6th International Conference of the CLEF Association, CLEF 2015, Proceedings
EditorsJosiane Mothe, Jacques Savoy, Jaap Kamps, Karen Pinel-Sauvagnat, Gareth Jones, Eric SanJuan, Linda Cappellato, Nicola Ferro
Place of PublicationCham, Switzerland
Number of pages15
ISBN (Electronic)9783319240275
ISBN (Print)9783319240268
Publication statusPublished - 2015
Event6th International Conference on Labs of the Evaluation Forum, CLEF 2015 - Toulouse, Toulouse, France
Duration: 8 Sept 201511 Sept 2015

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
ISSN (Print)03029743
ISSN (Electronic)16113349


Conference6th International Conference on Labs of the Evaluation Forum, CLEF 2015
Abbreviated titleCLEF 2015
OtherCLEF 2015 is the sixth CLEF conference continuing the popular CLEF campaigns which have run since 2000 contributing to the systematic evaluation of information access systems, primarily through experimentation on shared tasks.

Building on the format first introduced in 2010, CLEF 2015 consists of an independent peer-reviewed conference on a broad range of issues in the fields of multilingual and multimodal information access evaluation, and a set of labs and workshops designed to test different aspects of mono and cross-language Information retrieval systems. Together, the conference and the lab series will maintain and expand upon the CLEF tradition of community-based evaluation and discussion on evaluation issues
Internet address


Dive into the research topics of 'Overview of the CLEF eHealth evaluation lab 2015'. Together they form a unique fingerprint.

Cite this