Overview of the CLEF eHealth evaluation lab 2015

Lorraine Goeuriot, Liadh Kelly, Hanna Suominen, Leif Hanlen, Aurèlie Nèvèol, Cyril Grouin, João Palotti, Guido Zuccon

Research output: A Conference proceeding or a Chapter in BookConference contribution

47 Citations (Scopus)

Abstract

This paper reports on the 3rd CLEFeHealth evaluation lab, which continues our evaluation resource building activities for the medical domain. In this edition of the lab, we focus on easing patients and nurses in authoring, understanding, and accessing eHealth information. The 2015 CLEFeHealth evaluation lab was structured into two tasks, focusing on evaluating methods for information extraction (IE) and information retrieval (IR). The IE task introduced two new challenges. Task 1a focused on clinical speech recognition of nursing handover notes; Task 1b focused on clinical named entity recognition in languages other than English, specifically French. Task 2 focused on the retrieval of health information to answer queries issued by general consumers seeking information to understand their health symptoms or conditions. The number of teams registering their interest was 47 in Tasks 1 (2 teams in Task 1a and 7 teams in Task 1b) and 53 in Task 2 (12 teams) for a total of 20 unique teams. The best system recognized 4, 984 out of 6, 818 test words correctly and generated 2, 626 incorrect words (i.e., 38.5% error) in Task 1a; had the F-measure of 0.756 for plain entity recognition, 0.711 for normalized entity recognition, and 0.872 for entity normalization in Task 1b; and resulted in P@10 of 0.5394 and nDCG@10 of 0.5086 in Task 2. These results demonstrate the substantial community interest and capabilities of these systems in addressing challenges faced by patients and nurses. As in previous years, the organizers have made data and tools available for future research and development.

Original languageEnglish
Title of host publicationExperimental IR Meets Multilinguality, Multimodality, and Interaction
Subtitle of host publication6th International Conference of the CLEF Association, CLEF 2015, Proceedings
EditorsJosiane Mothe, Jacques Savoy, Jaap Kamps, Karen Pinel-Sauvagnat, Gareth Jones, Eric SanJuan, Linda Cappellato, Nicola Ferro
Place of PublicationCham, Switzerland
PublisherSpringer
Pages429-443
Number of pages15
Volume9283
ISBN (Electronic)9783319240275
ISBN (Print)9783319240268
DOIs
Publication statusPublished - 2015
Event6th International Conference on Labs of the Evaluation Forum, CLEF 2015 - Toulouse, Toulouse, France
Duration: 8 Sep 201511 Sep 2015
http://clef2015.clef-initiative.eu/publications.php

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9283
ISSN (Print)03029743
ISSN (Electronic)16113349

Conference

Conference6th International Conference on Labs of the Evaluation Forum, CLEF 2015
Abbreviated titleCLEF 2015
CountryFrance
CityToulouse
Period8/09/1511/09/15
OtherCLEF 2015 is the sixth CLEF conference continuing the popular CLEF campaigns which have run since 2000 contributing to the systematic evaluation of information access systems, primarily through experimentation on shared tasks.

Building on the format first introduced in 2010, CLEF 2015 consists of an independent peer-reviewed conference on a broad range of issues in the fields of multilingual and multimodal information access evaluation, and a set of labs and workshops designed to test different aspects of mono and cross-language Information retrieval systems. Together, the conference and the lab series will maintain and expand upon the CLEF tradition of community-based evaluation and discussion on evaluation issues
Internet address

    Fingerprint

Cite this

Goeuriot, L., Kelly, L., Suominen, H., Hanlen, L., Nèvèol, A., Grouin, C., ... Zuccon, G. (2015). Overview of the CLEF eHealth evaluation lab 2015. In J. Mothe, J. Savoy, J. Kamps, K. Pinel-Sauvagnat, G. Jones, E. SanJuan, L. Cappellato, ... N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction: 6th International Conference of the CLEF Association, CLEF 2015, Proceedings (Vol. 9283, pp. 429-443). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9283). Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-319-24027-5_44