Overview of the ShARe/CLEF eHealth evaluation lab 2014

Liadh Kelly, Lorraine Goeuriot, Hanna Suominen, Tobias Schreck, Gondy Leroy, Danielle L. Mowery, Sumithra Velupillai, Wendy W. Chapman, David Martinez, Guido Zuccon, João Palotti

Research output: A Conference proceeding or a Chapter in BookConference contribution

44 Citations (Scopus)

Abstract

This paper reports on the 2nd ShARe/CLEFeHealth evaluation lab which continues our evaluation resource building activities for the medical domain. In this lab we focus on patients' information needs as opposed to the more common campaign focus of the specialised information needs of physicians and other healthcare workers. The usage scenario of the lab is to ease patients and next-of-kins' ease in understanding eHealth information, in particular clinical reports. The 1st ShARe/CLEFeHealth evaluation lab was held in 2013. This lab consisted of three tasks. Task 1 focused on named entity recognition and normalization of disorders; Task 2 on normalization of acronyms/abbreviations; and Task 3 on information retrieval to address questions patients may have when reading clinical reports. This year's lab introduces a new challenge in Task 1 on visual-interactive search and exploration of eHealth data. Its aim is to help patients (or their next-of-kin) in readability issues related to their hospital discharge documents and related information search on the Internet. Task 2 then continues the information extraction work of the 2013 lab, specifically focusing on disorder attribute identification and normalization from clinical text. Finally, this year's Task 3 further extends the 2013 information retrieval task, by cleaning the 2013 document collection and introducing a new query generation method and multilingual queries. De-identified clinical reports used by the three tasks were from US intensive care and originated from the MIMIC II database. Other text documents for Tasks 1 and 3 were from the Internet and originated from the Khresmoi project. Task 2 annotations originated from the ShARe annotations. For Tasks 1 and 3, new annotations, queries, and relevance assessments were created. 50, 79, and 91 people registered their interest in Tasks 1, 2, and 3, respectively. 24 unique teams participated with 1, 10, and 14 teams in Tasks 1, 2 and 3, respectively. The teams were from Africa, Asia, Canada, Europe, and North America. The Task 1 submission, reviewed by 5 expert peers, related to the task evaluation category of Effective use of interaction and targeted the needs of both expert and novice users. The best system had an Accuracy of 0.868 in Task 2a, an F1-score of 0.576 in Task 2b, and Precision at 10 (P@10) of 0.756 in Task 3. The results demonstrate the substantial community interest and capabilities of these systems in making clinical reports easier to understand for patients. The organisers have made data and tools available for future research and development.

Original languageEnglish
Title of host publicationInformation Access Evaluation: Multilinguality, Multimodality, and Interaction
Subtitle of host publication5th International Conference of the CLEF Initiative, CLEF 2014, Proceedings
EditorsEvangelos Kanoulas, Mihai Lupu, Paul Clough, Mark Sanderson, Mark Hall, Allan Hanbury, Elaine Toms
Place of PublicationCham, Switzerland
PublisherSpringer
Pages172-191
Number of pages20
Volume8685
ISBN (Print)9783319113814
DOIs
Publication statusPublished - 2014
Event5th International Conference of the CLEF Initiative, CLEF 2014 - Sheffield, Sheffield, United Kingdom
Duration: 15 Sep 201418 Sep 2014

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8685 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference5th International Conference of the CLEF Initiative, CLEF 2014
Abbreviated titleCLEF 2014
CountryUnited Kingdom
CitySheffield
Period15/09/1418/09/14

Fingerprint

Information retrieval
Internet
Normalization
Annotation
Evaluation
Query
Information Retrieval
Disorder
Cleaning
Continue
Abbreviation
Named Entity Recognition
Acronym
Information Extraction
Research and Development
Healthcare
Attribute
Scenarios
Resources
Interaction

Cite this

Kelly, L., Goeuriot, L., Suominen, H., Schreck, T., Leroy, G., Mowery, D. L., ... Palotti, J. (2014). Overview of the ShARe/CLEF eHealth evaluation lab 2014. In E. Kanoulas, M. Lupu, P. Clough, M. Sanderson, M. Hall, A. Hanbury, & E. Toms (Eds.), Information Access Evaluation: Multilinguality, Multimodality, and Interaction: 5th International Conference of the CLEF Initiative, CLEF 2014, Proceedings (Vol. 8685, pp. 172-191). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8685 LNCS). Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-319-11382-1_17
Kelly, Liadh ; Goeuriot, Lorraine ; Suominen, Hanna ; Schreck, Tobias ; Leroy, Gondy ; Mowery, Danielle L. ; Velupillai, Sumithra ; Chapman, Wendy W. ; Martinez, David ; Zuccon, Guido ; Palotti, João. / Overview of the ShARe/CLEF eHealth evaluation lab 2014. Information Access Evaluation: Multilinguality, Multimodality, and Interaction: 5th International Conference of the CLEF Initiative, CLEF 2014, Proceedings. editor / Evangelos Kanoulas ; Mihai Lupu ; Paul Clough ; Mark Sanderson ; Mark Hall ; Allan Hanbury ; Elaine Toms. Vol. 8685 Cham, Switzerland : Springer, 2014. pp. 172-191 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{aa8dda563279463281d1f4abc93fba2e,
title = "Overview of the ShARe/CLEF eHealth evaluation lab 2014",
abstract = "This paper reports on the 2nd ShARe/CLEFeHealth evaluation lab which continues our evaluation resource building activities for the medical domain. In this lab we focus on patients' information needs as opposed to the more common campaign focus of the specialised information needs of physicians and other healthcare workers. The usage scenario of the lab is to ease patients and next-of-kins' ease in understanding eHealth information, in particular clinical reports. The 1st ShARe/CLEFeHealth evaluation lab was held in 2013. This lab consisted of three tasks. Task 1 focused on named entity recognition and normalization of disorders; Task 2 on normalization of acronyms/abbreviations; and Task 3 on information retrieval to address questions patients may have when reading clinical reports. This year's lab introduces a new challenge in Task 1 on visual-interactive search and exploration of eHealth data. Its aim is to help patients (or their next-of-kin) in readability issues related to their hospital discharge documents and related information search on the Internet. Task 2 then continues the information extraction work of the 2013 lab, specifically focusing on disorder attribute identification and normalization from clinical text. Finally, this year's Task 3 further extends the 2013 information retrieval task, by cleaning the 2013 document collection and introducing a new query generation method and multilingual queries. De-identified clinical reports used by the three tasks were from US intensive care and originated from the MIMIC II database. Other text documents for Tasks 1 and 3 were from the Internet and originated from the Khresmoi project. Task 2 annotations originated from the ShARe annotations. For Tasks 1 and 3, new annotations, queries, and relevance assessments were created. 50, 79, and 91 people registered their interest in Tasks 1, 2, and 3, respectively. 24 unique teams participated with 1, 10, and 14 teams in Tasks 1, 2 and 3, respectively. The teams were from Africa, Asia, Canada, Europe, and North America. The Task 1 submission, reviewed by 5 expert peers, related to the task evaluation category of Effective use of interaction and targeted the needs of both expert and novice users. The best system had an Accuracy of 0.868 in Task 2a, an F1-score of 0.576 in Task 2b, and Precision at 10 (P@10) of 0.756 in Task 3. The results demonstrate the substantial community interest and capabilities of these systems in making clinical reports easier to understand for patients. The organisers have made data and tools available for future research and development.",
keywords = "Evaluation, Information Extraction, Information Retrieval, Information Visualisation, Medical Informatics, Test-set Generation, Text Classification, Text Segmentation",
author = "Liadh Kelly and Lorraine Goeuriot and Hanna Suominen and Tobias Schreck and Gondy Leroy and Mowery, {Danielle L.} and Sumithra Velupillai and Chapman, {Wendy W.} and David Martinez and Guido Zuccon and Jo{\~a}o Palotti",
year = "2014",
doi = "10.1007/978-3-319-11382-1_17",
language = "English",
isbn = "9783319113814",
volume = "8685",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer",
pages = "172--191",
editor = "Evangelos Kanoulas and Mihai Lupu and Paul Clough and Mark Sanderson and Mark Hall and Allan Hanbury and Elaine Toms",
booktitle = "Information Access Evaluation: Multilinguality, Multimodality, and Interaction",
address = "Netherlands",

}

Kelly, L, Goeuriot, L, Suominen, H, Schreck, T, Leroy, G, Mowery, DL, Velupillai, S, Chapman, WW, Martinez, D, Zuccon, G & Palotti, J 2014, Overview of the ShARe/CLEF eHealth evaluation lab 2014. in E Kanoulas, M Lupu, P Clough, M Sanderson, M Hall, A Hanbury & E Toms (eds), Information Access Evaluation: Multilinguality, Multimodality, and Interaction: 5th International Conference of the CLEF Initiative, CLEF 2014, Proceedings. vol. 8685, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8685 LNCS, Springer, Cham, Switzerland, pp. 172-191, 5th International Conference of the CLEF Initiative, CLEF 2014, Sheffield, United Kingdom, 15/09/14. https://doi.org/10.1007/978-3-319-11382-1_17

Overview of the ShARe/CLEF eHealth evaluation lab 2014. / Kelly, Liadh; Goeuriot, Lorraine; Suominen, Hanna; Schreck, Tobias; Leroy, Gondy; Mowery, Danielle L.; Velupillai, Sumithra; Chapman, Wendy W.; Martinez, David; Zuccon, Guido; Palotti, João.

Information Access Evaluation: Multilinguality, Multimodality, and Interaction: 5th International Conference of the CLEF Initiative, CLEF 2014, Proceedings. ed. / Evangelos Kanoulas; Mihai Lupu; Paul Clough; Mark Sanderson; Mark Hall; Allan Hanbury; Elaine Toms. Vol. 8685 Cham, Switzerland : Springer, 2014. p. 172-191 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8685 LNCS).

Research output: A Conference proceeding or a Chapter in BookConference contribution

TY - GEN

T1 - Overview of the ShARe/CLEF eHealth evaluation lab 2014

AU - Kelly, Liadh

AU - Goeuriot, Lorraine

AU - Suominen, Hanna

AU - Schreck, Tobias

AU - Leroy, Gondy

AU - Mowery, Danielle L.

AU - Velupillai, Sumithra

AU - Chapman, Wendy W.

AU - Martinez, David

AU - Zuccon, Guido

AU - Palotti, João

PY - 2014

Y1 - 2014

N2 - This paper reports on the 2nd ShARe/CLEFeHealth evaluation lab which continues our evaluation resource building activities for the medical domain. In this lab we focus on patients' information needs as opposed to the more common campaign focus of the specialised information needs of physicians and other healthcare workers. The usage scenario of the lab is to ease patients and next-of-kins' ease in understanding eHealth information, in particular clinical reports. The 1st ShARe/CLEFeHealth evaluation lab was held in 2013. This lab consisted of three tasks. Task 1 focused on named entity recognition and normalization of disorders; Task 2 on normalization of acronyms/abbreviations; and Task 3 on information retrieval to address questions patients may have when reading clinical reports. This year's lab introduces a new challenge in Task 1 on visual-interactive search and exploration of eHealth data. Its aim is to help patients (or their next-of-kin) in readability issues related to their hospital discharge documents and related information search on the Internet. Task 2 then continues the information extraction work of the 2013 lab, specifically focusing on disorder attribute identification and normalization from clinical text. Finally, this year's Task 3 further extends the 2013 information retrieval task, by cleaning the 2013 document collection and introducing a new query generation method and multilingual queries. De-identified clinical reports used by the three tasks were from US intensive care and originated from the MIMIC II database. Other text documents for Tasks 1 and 3 were from the Internet and originated from the Khresmoi project. Task 2 annotations originated from the ShARe annotations. For Tasks 1 and 3, new annotations, queries, and relevance assessments were created. 50, 79, and 91 people registered their interest in Tasks 1, 2, and 3, respectively. 24 unique teams participated with 1, 10, and 14 teams in Tasks 1, 2 and 3, respectively. The teams were from Africa, Asia, Canada, Europe, and North America. The Task 1 submission, reviewed by 5 expert peers, related to the task evaluation category of Effective use of interaction and targeted the needs of both expert and novice users. The best system had an Accuracy of 0.868 in Task 2a, an F1-score of 0.576 in Task 2b, and Precision at 10 (P@10) of 0.756 in Task 3. The results demonstrate the substantial community interest and capabilities of these systems in making clinical reports easier to understand for patients. The organisers have made data and tools available for future research and development.

AB - This paper reports on the 2nd ShARe/CLEFeHealth evaluation lab which continues our evaluation resource building activities for the medical domain. In this lab we focus on patients' information needs as opposed to the more common campaign focus of the specialised information needs of physicians and other healthcare workers. The usage scenario of the lab is to ease patients and next-of-kins' ease in understanding eHealth information, in particular clinical reports. The 1st ShARe/CLEFeHealth evaluation lab was held in 2013. This lab consisted of three tasks. Task 1 focused on named entity recognition and normalization of disorders; Task 2 on normalization of acronyms/abbreviations; and Task 3 on information retrieval to address questions patients may have when reading clinical reports. This year's lab introduces a new challenge in Task 1 on visual-interactive search and exploration of eHealth data. Its aim is to help patients (or their next-of-kin) in readability issues related to their hospital discharge documents and related information search on the Internet. Task 2 then continues the information extraction work of the 2013 lab, specifically focusing on disorder attribute identification and normalization from clinical text. Finally, this year's Task 3 further extends the 2013 information retrieval task, by cleaning the 2013 document collection and introducing a new query generation method and multilingual queries. De-identified clinical reports used by the three tasks were from US intensive care and originated from the MIMIC II database. Other text documents for Tasks 1 and 3 were from the Internet and originated from the Khresmoi project. Task 2 annotations originated from the ShARe annotations. For Tasks 1 and 3, new annotations, queries, and relevance assessments were created. 50, 79, and 91 people registered their interest in Tasks 1, 2, and 3, respectively. 24 unique teams participated with 1, 10, and 14 teams in Tasks 1, 2 and 3, respectively. The teams were from Africa, Asia, Canada, Europe, and North America. The Task 1 submission, reviewed by 5 expert peers, related to the task evaluation category of Effective use of interaction and targeted the needs of both expert and novice users. The best system had an Accuracy of 0.868 in Task 2a, an F1-score of 0.576 in Task 2b, and Precision at 10 (P@10) of 0.756 in Task 3. The results demonstrate the substantial community interest and capabilities of these systems in making clinical reports easier to understand for patients. The organisers have made data and tools available for future research and development.

KW - Evaluation

KW - Information Extraction

KW - Information Retrieval

KW - Information Visualisation

KW - Medical Informatics

KW - Test-set Generation

KW - Text Classification

KW - Text Segmentation

UR - http://www.scopus.com/inward/record.url?scp=84906730141&partnerID=8YFLogxK

UR - http://www.mendeley.com/research/overview-shareclef-ehealth-evaluation-lab-2014

U2 - 10.1007/978-3-319-11382-1_17

DO - 10.1007/978-3-319-11382-1_17

M3 - Conference contribution

SN - 9783319113814

VL - 8685

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 172

EP - 191

BT - Information Access Evaluation: Multilinguality, Multimodality, and Interaction

A2 - Kanoulas, Evangelos

A2 - Lupu, Mihai

A2 - Clough, Paul

A2 - Sanderson, Mark

A2 - Hall, Mark

A2 - Hanbury, Allan

A2 - Toms, Elaine

PB - Springer

CY - Cham, Switzerland

ER -

Kelly L, Goeuriot L, Suominen H, Schreck T, Leroy G, Mowery DL et al. Overview of the ShARe/CLEF eHealth evaluation lab 2014. In Kanoulas E, Lupu M, Clough P, Sanderson M, Hall M, Hanbury A, Toms E, editors, Information Access Evaluation: Multilinguality, Multimodality, and Interaction: 5th International Conference of the CLEF Initiative, CLEF 2014, Proceedings. Vol. 8685. Cham, Switzerland: Springer. 2014. p. 172-191. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-11382-1_17