Overview of the CLEF eHealth evaluation lab 2016

Liadh Kelly; Lorraine Goeuriot; Hanna Suominen; Aurèlie Nèvèol; João Palotti; Guido Zuccon

doi:10.1007/978-3-319-44564-9_24

Overview of the CLEF eHealth evaluation lab 2016

Liadh Kelly, Lorraine Goeuriot, Hanna Suominen, Aurèlie Nèvèol, João Palotti, Guido Zuccon

Information Systems

Research output: A Conference proceeding or a Chapter in Book › Conference contribution › peer-review

61 Citations (Scopus)

Abstract

In this paper we provide an overview of the fourth edition of the CLEF eHealth evaluation lab. CLEF eHealth 2016 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins and clinical staff in understanding, accessing and authoring eHealth information in a multilingual setting. This year’s lab offered three tasks: Task 1 on handover information extraction related to Australian nursing shift changes, Task 2 on information extraction in French corpora, and Task 3 on multilingual patient-centred information retrieval considering query variations. In total 20 teams took part in these tasks (3 in Task 1, 7 in Task 2 and 10 in Task 3). Herein, we describe the resources created for these tasks, evaluation methodology adopted and provide a brief summary of participants to this year’s challenges and some results obtained. As in previous years, the organizers have made data and tools associated with the lab tasks available for future research and development.

Original language	English
Title of host publication	Experimental IR Meets Multilinguality, Multimodality, and Interaction
Subtitle of host publication	7th International Conference of the CLEF Association, CLEF 2016, Proceedings
Editors	Norbert Fuhr, Paulo Quaresma, Teresa Gonçalves, Birger Larsen, Krisztian Balog, Craig Macdonald, Linda Cappellato, Nicola Ferro
Place of Publication	Cham, Switzerland
Publisher	Springer
Pages	255-266
Number of pages	12
Volume	9822
ISBN (Electronic)	9783319445649
ISBN (Print)	9783319445632
DOIs	https://doi.org/10.1007/978-3-319-44564-9_24
Publication status	Published - 2016
Event	7th International Conference of the CLEF Association, CLEF 2016 - Evora, Evora, Portugal Duration: 5 Sept 2016 → 8 Sept 2016 http://clef2016.clef-initiative.eu/ (Conference website)

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	9822 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	7th International Conference of the CLEF Association, CLEF 2016
Abbreviated title	CLEF 2016
Country/Territory	Portugal
City	Evora
Period	5/09/16 → 8/09/16
Other	CLEF 2016 is the seventh CLEF conference continuing the popular CLEF campaigns which have run since 2000 contributing to the systematic evaluation of information access systems, primarily through experimentation on shared tasks. Building on the format first introduced in 2010, CLEF 2016 consists of an independent peer-reviewed conference on a broad range of issues in the fields of multilingual and multimodal information access evaluation, and a set of labs and workshops designed to test different aspects of mono and cross-language Information retrieval systems. Together, the conference and the lab series will maintain and expand upon the CLEF tradition of community-based evaluation and discussion on evaluation issues
Internet address	http://clef2016.clef-initiative.eu/ (Conference website)

Access to Document

10.1007/978-3-319-44564-9_24

Cite this

Kelly, L., Goeuriot, L., Suominen, H., Nèvèol, A., Palotti, J., & Zuccon, G. (2016). Overview of the CLEF eHealth evaluation lab 2016. In N. Fuhr, P. Quaresma, T. Gonçalves, B. Larsen, K. Balog, C. Macdonald, L. Cappellato, & N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction: 7th International Conference of the CLEF Association, CLEF 2016, Proceedings (Vol. 9822, pp. 255-266). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9822 LNCS). Springer. https://doi.org/10.1007/978-3-319-44564-9_24

Kelly, Liadh ; Goeuriot, Lorraine ; Suominen, Hanna et al. / Overview of the CLEF eHealth evaluation lab 2016. Experimental IR Meets Multilinguality, Multimodality, and Interaction: 7th International Conference of the CLEF Association, CLEF 2016, Proceedings. editor / Norbert Fuhr ; Paulo Quaresma ; Teresa Gonçalves ; Birger Larsen ; Krisztian Balog ; Craig Macdonald ; Linda Cappellato ; Nicola Ferro. Vol. 9822 Cham, Switzerland : Springer, 2016. pp. 255-266 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{8a4b8f0fcefa43059dc7d94fb63fc376,

title = "Overview of the CLEF eHealth evaluation lab 2016",

abstract = "In this paper we provide an overview of the fourth edition of the CLEF eHealth evaluation lab. CLEF eHealth 2016 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins and clinical staff in understanding, accessing and authoring eHealth information in a multilingual setting. This year{\textquoteright}s lab offered three tasks: Task 1 on handover information extraction related to Australian nursing shift changes, Task 2 on information extraction in French corpora, and Task 3 on multilingual patient-centred information retrieval considering query variations. In total 20 teams took part in these tasks (3 in Task 1, 7 in Task 2 and 10 in Task 3). Herein, we describe the resources created for these tasks, evaluation methodology adopted and provide a brief summary of participants to this year{\textquoteright}s challenges and some results obtained. As in previous years, the organizers have made data and tools associated with the lab tasks available for future research and development.",

keywords = "Entity linking, Evaluation, Information extraction, Information retrieval, Medical informatics, Nursing records, Patient handoff/handover, Self-diagnosis, Speech recognition, Test-set generation, Text classification, Text segmentation",

author = "Liadh Kelly and Lorraine Goeuriot and Hanna Suominen and Aur{\`e}lie N{\`e}v{\`e}ol and Jo{\~a}o Palotti and Guido Zuccon",

note = "Funding Information: We express our gratitude to Maricel Angel, Registered Nurse at NICTA, for helping us to create the Task 1 dataset, using the Prot{\'e}g{\'e} resource, which is supported by grant GM10331601 from the National Institute of General Medical Sciences of the United States National Institutes of Health. We are also thankful to the people involved in the query creation and relevance assessment exercise. Last but not least, we gratefully acknowledge the participating teams{\textquoteright} hard work. We thank them for their submissions and interest in the lab. Funding Information: The CLEF eHealth 2016 evaluation lab has been supported in part by (in alphabetical order) the ANR, the French National Research Agency, under grant CABeRneT ANR-13-JS02-0009-01; CLEF Initiative; ESF ELIAS network program Horizon 2020 program (H2020-ICT-2014-1) under grant agreement 644753 (KCONNECT); Microsoft Azure for Research Award CRM:0518649; NICTA, funded by the Australian Government through the Department of Communications and the Australian Research Council through the Information and Communications Technology (ICT) Centre of Excellence Program; and PhysioNetWorks Workspaces. Publisher Copyright: {\textcopyright} Springer International Publishing Switzerland 2016.; 7th International Conference of the CLEF Association, CLEF 2016, CLEF 2016 ; Conference date: 05-09-2016 Through 08-09-2016",

year = "2016",

doi = "10.1007/978-3-319-44564-9_24",

language = "English",

isbn = "9783319445632",

volume = "9822",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "255--266",

editor = "Norbert Fuhr and Paulo Quaresma and Teresa Gon{\c c}alves and Birger Larsen and Krisztian Balog and Craig Macdonald and Linda Cappellato and Nicola Ferro",

booktitle = "Experimental IR Meets Multilinguality, Multimodality, and Interaction",

address = "Netherlands",

url = "http://clef2016.clef-initiative.eu/",

}

Kelly, L, Goeuriot, L, Suominen, H, Nèvèol, A, Palotti, J & Zuccon, G 2016, Overview of the CLEF eHealth evaluation lab 2016. in N Fuhr, P Quaresma, T Gonçalves, B Larsen, K Balog, C Macdonald, L Cappellato & N Ferro (eds), Experimental IR Meets Multilinguality, Multimodality, and Interaction: 7th International Conference of the CLEF Association, CLEF 2016, Proceedings. vol. 9822, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9822 LNCS, Springer, Cham, Switzerland, pp. 255-266, 7th International Conference of the CLEF Association, CLEF 2016, Evora, Portugal, 5/09/16. https://doi.org/10.1007/978-3-319-44564-9_24

Overview of the CLEF eHealth evaluation lab 2016. / Kelly, Liadh; Goeuriot, Lorraine; Suominen, Hanna et al.
Experimental IR Meets Multilinguality, Multimodality, and Interaction: 7th International Conference of the CLEF Association, CLEF 2016, Proceedings. ed. / Norbert Fuhr; Paulo Quaresma; Teresa Gonçalves; Birger Larsen; Krisztian Balog; Craig Macdonald; Linda Cappellato; Nicola Ferro. Vol. 9822 Cham, Switzerland: Springer, 2016. p. 255-266 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9822 LNCS).

Research output: A Conference proceeding or a Chapter in Book › Conference contribution › peer-review

TY - GEN

T1 - Overview of the CLEF eHealth evaluation lab 2016

AU - Kelly, Liadh

AU - Goeuriot, Lorraine

AU - Suominen, Hanna

AU - Nèvèol, Aurèlie

AU - Palotti, João

AU - Zuccon, Guido

N1 - Funding Information: We express our gratitude to Maricel Angel, Registered Nurse at NICTA, for helping us to create the Task 1 dataset, using the Protégé resource, which is supported by grant GM10331601 from the National Institute of General Medical Sciences of the United States National Institutes of Health. We are also thankful to the people involved in the query creation and relevance assessment exercise. Last but not least, we gratefully acknowledge the participating teams’ hard work. We thank them for their submissions and interest in the lab. Funding Information: The CLEF eHealth 2016 evaluation lab has been supported in part by (in alphabetical order) the ANR, the French National Research Agency, under grant CABeRneT ANR-13-JS02-0009-01; CLEF Initiative; ESF ELIAS network program Horizon 2020 program (H2020-ICT-2014-1) under grant agreement 644753 (KCONNECT); Microsoft Azure for Research Award CRM:0518649; NICTA, funded by the Australian Government through the Department of Communications and the Australian Research Council through the Information and Communications Technology (ICT) Centre of Excellence Program; and PhysioNetWorks Workspaces. Publisher Copyright: © Springer International Publishing Switzerland 2016.

PY - 2016

Y1 - 2016

N2 - In this paper we provide an overview of the fourth edition of the CLEF eHealth evaluation lab. CLEF eHealth 2016 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins and clinical staff in understanding, accessing and authoring eHealth information in a multilingual setting. This year’s lab offered three tasks: Task 1 on handover information extraction related to Australian nursing shift changes, Task 2 on information extraction in French corpora, and Task 3 on multilingual patient-centred information retrieval considering query variations. In total 20 teams took part in these tasks (3 in Task 1, 7 in Task 2 and 10 in Task 3). Herein, we describe the resources created for these tasks, evaluation methodology adopted and provide a brief summary of participants to this year’s challenges and some results obtained. As in previous years, the organizers have made data and tools associated with the lab tasks available for future research and development.

AB - In this paper we provide an overview of the fourth edition of the CLEF eHealth evaluation lab. CLEF eHealth 2016 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins and clinical staff in understanding, accessing and authoring eHealth information in a multilingual setting. This year’s lab offered three tasks: Task 1 on handover information extraction related to Australian nursing shift changes, Task 2 on information extraction in French corpora, and Task 3 on multilingual patient-centred information retrieval considering query variations. In total 20 teams took part in these tasks (3 in Task 1, 7 in Task 2 and 10 in Task 3). Herein, we describe the resources created for these tasks, evaluation methodology adopted and provide a brief summary of participants to this year’s challenges and some results obtained. As in previous years, the organizers have made data and tools associated with the lab tasks available for future research and development.

KW - Entity linking

KW - Evaluation

KW - Information extraction

KW - Information retrieval

KW - Medical informatics

KW - Nursing records

KW - Patient handoff/handover

KW - Self-diagnosis

KW - Speech recognition

KW - Test-set generation

KW - Text classification

KW - Text segmentation

UR - http://www.scopus.com/inward/record.url?scp=84984804764&partnerID=8YFLogxK

UR - http://www.mendeley.com/research/overview-clef-ehealth-evaluation-lab-2016

U2 - 10.1007/978-3-319-44564-9_24

DO - 10.1007/978-3-319-44564-9_24

M3 - Conference contribution

AN - SCOPUS:84984804764

SN - 9783319445632

VL - 9822

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 255

EP - 266

BT - Experimental IR Meets Multilinguality, Multimodality, and Interaction

A2 - Fuhr, Norbert

A2 - Quaresma, Paulo

A2 - Gonçalves, Teresa

A2 - Larsen, Birger

A2 - Balog, Krisztian

A2 - Macdonald, Craig

A2 - Cappellato, Linda

A2 - Ferro, Nicola

PB - Springer

CY - Cham, Switzerland

T2 - 7th International Conference of the CLEF Association, CLEF 2016

Y2 - 5 September 2016 through 8 September 2016

ER -

Kelly L, Goeuriot L, Suominen H, Nèvèol A, Palotti J, Zuccon G. Overview of the CLEF eHealth evaluation lab 2016. In Fuhr N, Quaresma P, Gonçalves T, Larsen B, Balog K, Macdonald C, Cappellato L, Ferro N, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction: 7th International Conference of the CLEF Association, CLEF 2016, Proceedings. Vol. 9822. Cham, Switzerland: Springer. 2016. p. 255-266. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-44564-9_24

Overview of the CLEF eHealth evaluation lab 2016

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this