Abstract
This paper reports on the 3rd CLEFeHealth evaluation lab, which continues our evaluation resource building activities for the medical domain. In this edition of the lab, we focus on easing patients and nurses in authoring, understanding, and accessing eHealth information. The 2015 CLEFeHealth evaluation lab was structured into two tasks, focusing on evaluating methods for information extraction (IE) and information retrieval (IR). The IE task introduced two new challenges. Task 1a focused on clinical speech recognition of nursing handover notes; Task 1b focused on clinical named entity recognition in languages other than English, specifically French. Task 2 focused on the retrieval of health information to answer queries issued by general consumers seeking information to understand their health symptoms or conditions. The number of teams registering their interest was 47 in Tasks 1 (2 teams in Task 1a and 7 teams in Task 1b) and 53 in Task 2 (12 teams) for a total of 20 unique teams. The best system recognized 4, 984 out of 6, 818 test words correctly and generated 2, 626 incorrect words (i.e., 38.5% error) in Task 1a; had the F-measure of 0.756 for plain entity recognition, 0.711 for normalized entity recognition, and 0.872 for entity normalization in Task 1b; and resulted in P@10 of 0.5394 and nDCG@10 of 0.5086 in Task 2. These results demonstrate the substantial community interest and capabilities of these systems in addressing challenges faced by patients and nurses. As in previous years, the organizers have made data and tools available for future research and development.
| Original language | English |
|---|---|
| Title of host publication | Experimental IR Meets Multilinguality, Multimodality, and Interaction |
| Subtitle of host publication | 6th International Conference of the CLEF Association, CLEF 2015, Proceedings |
| Editors | Josiane Mothe, Jacques Savoy, Jaap Kamps, Karen Pinel-Sauvagnat, Gareth Jones, Eric SanJuan, Linda Cappellato, Nicola Ferro |
| Place of Publication | Cham, Switzerland |
| Publisher | Springer |
| Pages | 429-443 |
| Number of pages | 15 |
| Volume | 9283 |
| ISBN (Electronic) | 9783319240275 |
| ISBN (Print) | 9783319240268 |
| DOIs | |
| Publication status | Published - 2015 |
| Event | 6th International Conference on Labs of the Evaluation Forum, CLEF 2015 - Toulouse, Toulouse, France Duration: 8 Sept 2015 → 11 Sept 2015 http://clef2015.clef-initiative.eu/publications.php |
Publication series
| Name | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
|---|---|
| Volume | 9283 |
| ISSN (Print) | 03029743 |
| ISSN (Electronic) | 16113349 |
Conference
| Conference | 6th International Conference on Labs of the Evaluation Forum, CLEF 2015 |
|---|---|
| Abbreviated title | CLEF 2015 |
| Country/Territory | France |
| City | Toulouse |
| Period | 8/09/15 → 11/09/15 |
| Other | CLEF 2015 is the sixth CLEF conference continuing the popular CLEF campaigns which have run since 2000 contributing to the systematic evaluation of information access systems, primarily through experimentation on shared tasks. Building on the format first introduced in 2010, CLEF 2015 consists of an independent peer-reviewed conference on a broad range of issues in the fields of multilingual and multimodal information access evaluation, and a set of labs and workshops designed to test different aspects of mono and cross-language Information retrieval systems. Together, the conference and the lab series will maintain and expand upon the CLEF tradition of community-based evaluation and discussion on evaluation issues |
| Internet address |