Evaluating the state of the art in disorder recognition and normalization of the clinical narrative

Sameer Pradhan, Noémie Elhadad, Brett R. South, David Martínez, Lee Christensen, Amy Vogel, Hanna Suominen, Wendy W. Chapman, Guergana Savova

    Research output: Contribution to journalArticle

    48 Citations (Scopus)
    1 Downloads (Pure)

    Abstract

    Objective: The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art on the clinical text in (i) disorder mention identification/recognition based on Unified Medical Language System (UMLS) definition (Task 1a) and (ii) disorder mention normalization to an ontology (Task 1b). Such a community evaluation has not been previously executed. Task 1a included a total of 22 system submissions, and Task 1b included 17. Most of the systems employed a combination of rules and machine learners. Materials and methods: We used a subset of the Shared Annotated Resources (ShARe) corpus of annotated clinical text-199 clinical notes for training and 99 for testing (roughly 180 K words in total). We provided the community with the annotated gold standard training documents to build systems to identify and normalize disorder mentions. The systems were tested on a held-out gold standard test set to measure their performance. Results: For Task 1a, the best-performing system achieved an F1 score of 0.75 (0.80 precision; 0.71 recall). For Task 1b, another system performed best with an accuracy of 0.59. Discussion Most of the participating systems used a hybrid approach by supplementing machine-learning algorithms with features generated by rules and gazetteers created from the training data and from external resources. Conclusions: The task of disorder normalization is more challenging than that of identification. The ShARe corpus is available to the community as a reference standard for future studies.
    Original languageEnglish
    Pages (from-to)143-154
    Number of pages12
    JournalJournal of the American Medical Informatics Association
    Volume22
    Issue number1
    DOIs
    Publication statusPublished - 2014

    Fingerprint

    Unified Medical Language System
    Telemedicine
    Machine Learning

    Cite this

    Pradhan, Sameer ; Elhadad, Noémie ; South, Brett R. ; Martínez, David ; Christensen, Lee ; Vogel, Amy ; Suominen, Hanna ; Chapman, Wendy W. ; Savova, Guergana. / Evaluating the state of the art in disorder recognition and normalization of the clinical narrative. In: Journal of the American Medical Informatics Association. 2014 ; Vol. 22, No. 1. pp. 143-154.
    @article{2c85b7cb027a432aa4f2ac2e6c38e29e,
    title = "Evaluating the state of the art in disorder recognition and normalization of the clinical narrative",
    abstract = "Objective: The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art on the clinical text in (i) disorder mention identification/recognition based on Unified Medical Language System (UMLS) definition (Task 1a) and (ii) disorder mention normalization to an ontology (Task 1b). Such a community evaluation has not been previously executed. Task 1a included a total of 22 system submissions, and Task 1b included 17. Most of the systems employed a combination of rules and machine learners. Materials and methods: We used a subset of the Shared Annotated Resources (ShARe) corpus of annotated clinical text-199 clinical notes for training and 99 for testing (roughly 180 K words in total). We provided the community with the annotated gold standard training documents to build systems to identify and normalize disorder mentions. The systems were tested on a held-out gold standard test set to measure their performance. Results: For Task 1a, the best-performing system achieved an F1 score of 0.75 (0.80 precision; 0.71 recall). For Task 1b, another system performed best with an accuracy of 0.59. Discussion Most of the participating systems used a hybrid approach by supplementing machine-learning algorithms with features generated by rules and gazetteers created from the training data and from external resources. Conclusions: The task of disorder normalization is more challenging than that of identification. The ShARe corpus is available to the community as a reference standard for future studies.",
    keywords = "Biological Ontologies, Controlled, Datasets as Topic, Disease, Electronic Health Records, Humans, Information Storage and Retrieval, Information Storage and Retrieval: methods, Natural Language Processing, Systematized Nomenclature of Medicine, Unified Medical Language System, Vocabulary",
    author = "Sameer Pradhan and No{\'e}mie Elhadad and South, {Brett R.} and David Mart{\'i}nez and Lee Christensen and Amy Vogel and Hanna Suominen and Chapman, {Wendy W.} and Guergana Savova",
    year = "2014",
    doi = "10.1136/amiajnl-2013-002544",
    language = "English",
    volume = "22",
    pages = "143--154",
    journal = "Journal of the American Medical Informatics Association : JAMIA",
    issn = "1067-5027",
    publisher = "Oxford University Press",
    number = "1",

    }

    Pradhan, S, Elhadad, N, South, BR, Martínez, D, Christensen, L, Vogel, A, Suominen, H, Chapman, WW & Savova, G 2014, 'Evaluating the state of the art in disorder recognition and normalization of the clinical narrative', Journal of the American Medical Informatics Association, vol. 22, no. 1, pp. 143-154. https://doi.org/10.1136/amiajnl-2013-002544

    Evaluating the state of the art in disorder recognition and normalization of the clinical narrative. / Pradhan, Sameer; Elhadad, Noémie; South, Brett R.; Martínez, David; Christensen, Lee; Vogel, Amy; Suominen, Hanna; Chapman, Wendy W.; Savova, Guergana.

    In: Journal of the American Medical Informatics Association, Vol. 22, No. 1, 2014, p. 143-154.

    Research output: Contribution to journalArticle

    TY - JOUR

    T1 - Evaluating the state of the art in disorder recognition and normalization of the clinical narrative

    AU - Pradhan, Sameer

    AU - Elhadad, Noémie

    AU - South, Brett R.

    AU - Martínez, David

    AU - Christensen, Lee

    AU - Vogel, Amy

    AU - Suominen, Hanna

    AU - Chapman, Wendy W.

    AU - Savova, Guergana

    PY - 2014

    Y1 - 2014

    N2 - Objective: The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art on the clinical text in (i) disorder mention identification/recognition based on Unified Medical Language System (UMLS) definition (Task 1a) and (ii) disorder mention normalization to an ontology (Task 1b). Such a community evaluation has not been previously executed. Task 1a included a total of 22 system submissions, and Task 1b included 17. Most of the systems employed a combination of rules and machine learners. Materials and methods: We used a subset of the Shared Annotated Resources (ShARe) corpus of annotated clinical text-199 clinical notes for training and 99 for testing (roughly 180 K words in total). We provided the community with the annotated gold standard training documents to build systems to identify and normalize disorder mentions. The systems were tested on a held-out gold standard test set to measure their performance. Results: For Task 1a, the best-performing system achieved an F1 score of 0.75 (0.80 precision; 0.71 recall). For Task 1b, another system performed best with an accuracy of 0.59. Discussion Most of the participating systems used a hybrid approach by supplementing machine-learning algorithms with features generated by rules and gazetteers created from the training data and from external resources. Conclusions: The task of disorder normalization is more challenging than that of identification. The ShARe corpus is available to the community as a reference standard for future studies.

    AB - Objective: The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art on the clinical text in (i) disorder mention identification/recognition based on Unified Medical Language System (UMLS) definition (Task 1a) and (ii) disorder mention normalization to an ontology (Task 1b). Such a community evaluation has not been previously executed. Task 1a included a total of 22 system submissions, and Task 1b included 17. Most of the systems employed a combination of rules and machine learners. Materials and methods: We used a subset of the Shared Annotated Resources (ShARe) corpus of annotated clinical text-199 clinical notes for training and 99 for testing (roughly 180 K words in total). We provided the community with the annotated gold standard training documents to build systems to identify and normalize disorder mentions. The systems were tested on a held-out gold standard test set to measure their performance. Results: For Task 1a, the best-performing system achieved an F1 score of 0.75 (0.80 precision; 0.71 recall). For Task 1b, another system performed best with an accuracy of 0.59. Discussion Most of the participating systems used a hybrid approach by supplementing machine-learning algorithms with features generated by rules and gazetteers created from the training data and from external resources. Conclusions: The task of disorder normalization is more challenging than that of identification. The ShARe corpus is available to the community as a reference standard for future studies.

    KW - Biological Ontologies

    KW - Controlled

    KW - Datasets as Topic

    KW - Disease

    KW - Electronic Health Records

    KW - Humans

    KW - Information Storage and Retrieval

    KW - Information Storage and Retrieval: methods

    KW - Natural Language Processing

    KW - Systematized Nomenclature of Medicine

    KW - Unified Medical Language System

    KW - Vocabulary

    U2 - 10.1136/amiajnl-2013-002544

    DO - 10.1136/amiajnl-2013-002544

    M3 - Article

    VL - 22

    SP - 143

    EP - 154

    JO - Journal of the American Medical Informatics Association : JAMIA

    JF - Journal of the American Medical Informatics Association : JAMIA

    SN - 1067-5027

    IS - 1

    ER -