Overview of the ShARe/CLEF eHealth evaluation lab 2013

Hanna Suominen, Sanna Salanterä, Sumithra Velupillai, Wendy W. Chapman, Guergana Savova, Noemie Elhadad, Sameer Pradhan, Brett R. South, Danielle L. Mowery, Gareth J.F. Jones, Johannes Leveling, Liadh Kelly, Lorraine Goeuriot, David Martinez, Guido Zuccon

Research output: A Conference proceeding or a Chapter in BookConference contribution

100 Citations (Scopus)

Abstract

Discharge summaries and other free-text reports in healthcare transfer information between working shifts and geographic locations. Patients are likely to have difficulties in understanding their content, because of their medical jargon, non-standard abbreviations, and ward-specific idioms. This paper reports on an evaluation lab with an aim to support the continuum of care by developing methods and resources that make clinical reports in English easier to understand for patients, and which helps them in finding information related to their condition. This ShARe/CLEFeHealth2013 lab offered student mentoring and shared tasks: identification and normalisation of disorders (1a and 1b) and normalisation of abbreviations and acronyms (2) in clinical reports with respect to terminology standards in healthcare as well as information retrieval (3) to address questions patients may have when reading clinical reports. The focus on patients' information needs as opposed to the specialised information needs of physicians and other healthcare workers was the main feature of the lab distinguishing it from previous shared tasks. De-identified clinical reports for the three tasks were from US intensive care and originated from the MIMIC II database. Other text documents for Task 3 were from the Internet and originated from the Khresmoi project. Task 1 annotations originated from the ShARe annotations. For Tasks 2 and 3, new annotations, queries, and relevance assessments were created. 64, 56, and 55 people registered their interest in Tasks 1, 2, and 3, respectively. 34 unique teams (3 members per team on average) participated with 22, 17, 5, and 9 teams in Tasks 1a, 1b, 2 and 3, respectively. The teams were from Australia, China, France, India, Ireland, Republic of Korea, Spain, UK, and USA. Some teams developed and used additional annotations, but this strategy contributed to the system performance only in Task 2. The best systems had the F1 score of 0.75 in Task 1a; Accuracies of 0.59 and 0.72 in Tasks 1b and 2; and Precision at 10 of 0.52 in Task 3. The results demonstrate the substantial community interest and capabilities of these systems in making clinical reports easier to understand for patients. The organisers have made data and tools available for future research and development.

Original languageEnglish
Title of host publicationInformation Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings
Subtitle of host publicationLecture Notes in Computer Science
EditorsP Forner, H. Muller, R Paredes, P. Rosso, B. Stein
Place of PublicationGermany
PublisherSpringer
Pages212-231
Number of pages20
Volume8138
ISBN (Print)9783642408014
DOIs
Publication statusPublished - 2013
Externally publishedYes
Event4th International Conference of the CLEF Initiative, CLEF 2013 - Valencia, Valencia, Spain
Duration: 23 Sep 201326 Sep 2013

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8138 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Conference

Conference4th International Conference of the CLEF Initiative, CLEF 2013
CountrySpain
CityValencia
Period23/09/1326/09/13

Fingerprint

Terminology
Information retrieval
Internet
Annotation
Students
Healthcare
Evaluation
Abbreviation
Normalization
Acronym
Information Transfer
India
Research and Development
Information Retrieval
Disorder
System Performance
China
Continuum
Likely
Query

Cite this

Suominen, H., Salanterä, S., Velupillai, S., Chapman, W. W., Savova, G., Elhadad, N., ... Zuccon, G. (2013). Overview of the ShARe/CLEF eHealth evaluation lab 2013. In P. Forner, H. Muller, R. Paredes, P. Rosso, & B. Stein (Eds.), Information Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings: Lecture Notes in Computer Science (Vol. 8138, pp. 212-231). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8138 LNCS). Germany: Springer. https://doi.org/10.1007/978-3-642-40802-1_24
Suominen, Hanna ; Salanterä, Sanna ; Velupillai, Sumithra ; Chapman, Wendy W. ; Savova, Guergana ; Elhadad, Noemie ; Pradhan, Sameer ; South, Brett R. ; Mowery, Danielle L. ; Jones, Gareth J.F. ; Leveling, Johannes ; Kelly, Liadh ; Goeuriot, Lorraine ; Martinez, David ; Zuccon, Guido. / Overview of the ShARe/CLEF eHealth evaluation lab 2013. Information Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings: Lecture Notes in Computer Science. editor / P Forner ; H. Muller ; R Paredes ; P. Rosso ; B. Stein. Vol. 8138 Germany : Springer, 2013. pp. 212-231 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{44da56af8c6c4e3e87ba68c3caea3a49,
title = "Overview of the ShARe/CLEF eHealth evaluation lab 2013",
abstract = "Discharge summaries and other free-text reports in healthcare transfer information between working shifts and geographic locations. Patients are likely to have difficulties in understanding their content, because of their medical jargon, non-standard abbreviations, and ward-specific idioms. This paper reports on an evaluation lab with an aim to support the continuum of care by developing methods and resources that make clinical reports in English easier to understand for patients, and which helps them in finding information related to their condition. This ShARe/CLEFeHealth2013 lab offered student mentoring and shared tasks: identification and normalisation of disorders (1a and 1b) and normalisation of abbreviations and acronyms (2) in clinical reports with respect to terminology standards in healthcare as well as information retrieval (3) to address questions patients may have when reading clinical reports. The focus on patients' information needs as opposed to the specialised information needs of physicians and other healthcare workers was the main feature of the lab distinguishing it from previous shared tasks. De-identified clinical reports for the three tasks were from US intensive care and originated from the MIMIC II database. Other text documents for Task 3 were from the Internet and originated from the Khresmoi project. Task 1 annotations originated from the ShARe annotations. For Tasks 2 and 3, new annotations, queries, and relevance assessments were created. 64, 56, and 55 people registered their interest in Tasks 1, 2, and 3, respectively. 34 unique teams (3 members per team on average) participated with 22, 17, 5, and 9 teams in Tasks 1a, 1b, 2 and 3, respectively. The teams were from Australia, China, France, India, Ireland, Republic of Korea, Spain, UK, and USA. Some teams developed and used additional annotations, but this strategy contributed to the system performance only in Task 2. The best systems had the F1 score of 0.75 in Task 1a; Accuracies of 0.59 and 0.72 in Tasks 1b and 2; and Precision at 10 of 0.52 in Task 3. The results demonstrate the substantial community interest and capabilities of these systems in making clinical reports easier to understand for patients. The organisers have made data and tools available for future research and development.",
keywords = "Evaluation, Information Retrieval, Medical Informatics, Test-set Generation, Text Classification, Text Segmentation",
author = "Hanna Suominen and Sanna Salanter{\"a} and Sumithra Velupillai and Chapman, {Wendy W.} and Guergana Savova and Noemie Elhadad and Sameer Pradhan and South, {Brett R.} and Mowery, {Danielle L.} and Jones, {Gareth J.F.} and Johannes Leveling and Liadh Kelly and Lorraine Goeuriot and David Martinez and Guido Zuccon",
year = "2013",
doi = "10.1007/978-3-642-40802-1_24",
language = "English",
isbn = "9783642408014",
volume = "8138",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer",
pages = "212--231",
editor = "P Forner and H. Muller and R Paredes and P. Rosso and B. Stein",
booktitle = "Information Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings",
address = "Netherlands",

}

Suominen, H, Salanterä, S, Velupillai, S, Chapman, WW, Savova, G, Elhadad, N, Pradhan, S, South, BR, Mowery, DL, Jones, GJF, Leveling, J, Kelly, L, Goeuriot, L, Martinez, D & Zuccon, G 2013, Overview of the ShARe/CLEF eHealth evaluation lab 2013. in P Forner, H Muller, R Paredes, P Rosso & B Stein (eds), Information Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings: Lecture Notes in Computer Science. vol. 8138, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8138 LNCS, Springer, Germany, pp. 212-231, 4th International Conference of the CLEF Initiative, CLEF 2013, Valencia, Spain, 23/09/13. https://doi.org/10.1007/978-3-642-40802-1_24

Overview of the ShARe/CLEF eHealth evaluation lab 2013. / Suominen, Hanna; Salanterä, Sanna; Velupillai, Sumithra; Chapman, Wendy W.; Savova, Guergana; Elhadad, Noemie; Pradhan, Sameer; South, Brett R.; Mowery, Danielle L.; Jones, Gareth J.F.; Leveling, Johannes; Kelly, Liadh; Goeuriot, Lorraine; Martinez, David; Zuccon, Guido.

Information Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings: Lecture Notes in Computer Science. ed. / P Forner; H. Muller; R Paredes; P. Rosso; B. Stein. Vol. 8138 Germany : Springer, 2013. p. 212-231 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8138 LNCS).

Research output: A Conference proceeding or a Chapter in BookConference contribution

TY - GEN

T1 - Overview of the ShARe/CLEF eHealth evaluation lab 2013

AU - Suominen, Hanna

AU - Salanterä, Sanna

AU - Velupillai, Sumithra

AU - Chapman, Wendy W.

AU - Savova, Guergana

AU - Elhadad, Noemie

AU - Pradhan, Sameer

AU - South, Brett R.

AU - Mowery, Danielle L.

AU - Jones, Gareth J.F.

AU - Leveling, Johannes

AU - Kelly, Liadh

AU - Goeuriot, Lorraine

AU - Martinez, David

AU - Zuccon, Guido

PY - 2013

Y1 - 2013

N2 - Discharge summaries and other free-text reports in healthcare transfer information between working shifts and geographic locations. Patients are likely to have difficulties in understanding their content, because of their medical jargon, non-standard abbreviations, and ward-specific idioms. This paper reports on an evaluation lab with an aim to support the continuum of care by developing methods and resources that make clinical reports in English easier to understand for patients, and which helps them in finding information related to their condition. This ShARe/CLEFeHealth2013 lab offered student mentoring and shared tasks: identification and normalisation of disorders (1a and 1b) and normalisation of abbreviations and acronyms (2) in clinical reports with respect to terminology standards in healthcare as well as information retrieval (3) to address questions patients may have when reading clinical reports. The focus on patients' information needs as opposed to the specialised information needs of physicians and other healthcare workers was the main feature of the lab distinguishing it from previous shared tasks. De-identified clinical reports for the three tasks were from US intensive care and originated from the MIMIC II database. Other text documents for Task 3 were from the Internet and originated from the Khresmoi project. Task 1 annotations originated from the ShARe annotations. For Tasks 2 and 3, new annotations, queries, and relevance assessments were created. 64, 56, and 55 people registered their interest in Tasks 1, 2, and 3, respectively. 34 unique teams (3 members per team on average) participated with 22, 17, 5, and 9 teams in Tasks 1a, 1b, 2 and 3, respectively. The teams were from Australia, China, France, India, Ireland, Republic of Korea, Spain, UK, and USA. Some teams developed and used additional annotations, but this strategy contributed to the system performance only in Task 2. The best systems had the F1 score of 0.75 in Task 1a; Accuracies of 0.59 and 0.72 in Tasks 1b and 2; and Precision at 10 of 0.52 in Task 3. The results demonstrate the substantial community interest and capabilities of these systems in making clinical reports easier to understand for patients. The organisers have made data and tools available for future research and development.

AB - Discharge summaries and other free-text reports in healthcare transfer information between working shifts and geographic locations. Patients are likely to have difficulties in understanding their content, because of their medical jargon, non-standard abbreviations, and ward-specific idioms. This paper reports on an evaluation lab with an aim to support the continuum of care by developing methods and resources that make clinical reports in English easier to understand for patients, and which helps them in finding information related to their condition. This ShARe/CLEFeHealth2013 lab offered student mentoring and shared tasks: identification and normalisation of disorders (1a and 1b) and normalisation of abbreviations and acronyms (2) in clinical reports with respect to terminology standards in healthcare as well as information retrieval (3) to address questions patients may have when reading clinical reports. The focus on patients' information needs as opposed to the specialised information needs of physicians and other healthcare workers was the main feature of the lab distinguishing it from previous shared tasks. De-identified clinical reports for the three tasks were from US intensive care and originated from the MIMIC II database. Other text documents for Task 3 were from the Internet and originated from the Khresmoi project. Task 1 annotations originated from the ShARe annotations. For Tasks 2 and 3, new annotations, queries, and relevance assessments were created. 64, 56, and 55 people registered their interest in Tasks 1, 2, and 3, respectively. 34 unique teams (3 members per team on average) participated with 22, 17, 5, and 9 teams in Tasks 1a, 1b, 2 and 3, respectively. The teams were from Australia, China, France, India, Ireland, Republic of Korea, Spain, UK, and USA. Some teams developed and used additional annotations, but this strategy contributed to the system performance only in Task 2. The best systems had the F1 score of 0.75 in Task 1a; Accuracies of 0.59 and 0.72 in Tasks 1b and 2; and Precision at 10 of 0.52 in Task 3. The results demonstrate the substantial community interest and capabilities of these systems in making clinical reports easier to understand for patients. The organisers have made data and tools available for future research and development.

KW - Evaluation

KW - Information Retrieval

KW - Medical Informatics

KW - Test-set Generation

KW - Text Classification

KW - Text Segmentation

UR - http://www.scopus.com/inward/record.url?scp=84886418080&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-40802-1_24

DO - 10.1007/978-3-642-40802-1_24

M3 - Conference contribution

SN - 9783642408014

VL - 8138

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 212

EP - 231

BT - Information Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings

A2 - Forner, P

A2 - Muller, H.

A2 - Paredes, R

A2 - Rosso, P.

A2 - Stein, B.

PB - Springer

CY - Germany

ER -

Suominen H, Salanterä S, Velupillai S, Chapman WW, Savova G, Elhadad N et al. Overview of the ShARe/CLEF eHealth evaluation lab 2013. In Forner P, Muller H, Paredes R, Rosso P, Stein B, editors, Information Access Evaluation: Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Proceedings: Lecture Notes in Computer Science. Vol. 8138. Germany: Springer. 2013. p. 212-231. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-642-40802-1_24