Evaluating the state of the art in disorder recognition and normalization of the clinical narrative

Sameer Pradhan, Noémie Elhadad, Brett R. South, David Martínez, Lee Christensen, Amy Vogel, Hanna Suominen, Wendy W. Chapman, Guergana Savova

    Research output: Contribution to journalArticle

    56 Citations (Scopus)
    5 Downloads (Pure)

    Abstract

    Objective: The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art on the clinical text in (i) disorder mention identification/recognition based on Unified Medical Language System (UMLS) definition (Task 1a) and (ii) disorder mention normalization to an ontology (Task 1b). Such a community evaluation has not been previously executed. Task 1a included a total of 22 system submissions, and Task 1b included 17. Most of the systems employed a combination of rules and machine learners. Materials and methods: We used a subset of the Shared Annotated Resources (ShARe) corpus of annotated clinical text-199 clinical notes for training and 99 for testing (roughly 180 K words in total). We provided the community with the annotated gold standard training documents to build systems to identify and normalize disorder mentions. The systems were tested on a held-out gold standard test set to measure their performance. Results: For Task 1a, the best-performing system achieved an F1 score of 0.75 (0.80 precision; 0.71 recall). For Task 1b, another system performed best with an accuracy of 0.59. Discussion Most of the participating systems used a hybrid approach by supplementing machine-learning algorithms with features generated by rules and gazetteers created from the training data and from external resources. Conclusions: The task of disorder normalization is more challenging than that of identification. The ShARe corpus is available to the community as a reference standard for future studies.
    Original languageEnglish
    Pages (from-to)143-154
    Number of pages12
    JournalJournal of the American Medical Informatics Association
    Volume22
    Issue number1
    DOIs
    Publication statusPublished - 2014

    Fingerprint Dive into the research topics of 'Evaluating the state of the art in disorder recognition and normalization of the clinical narrative'. Together they form a unique fingerprint.

  • Cite this

    Pradhan, S., Elhadad, N., South, B. R., Martínez, D., Christensen, L., Vogel, A., Suominen, H., Chapman, W. W., & Savova, G. (2014). Evaluating the state of the art in disorder recognition and normalization of the clinical narrative. Journal of the American Medical Informatics Association, 22(1), 143-154. https://doi.org/10.1136/amiajnl-2013-002544