This Explains That: Congruent Image–Report Generation for Explainable Medical Image Analysis with Cyclic Generative Adversarial Networks

Abhineet Pandey, Bhawna Paliwal, Abhinav Dhall, Ramanathan Subramanian, Dwarikanath Mahapatra

Research output: A Conference proceeding or a Chapter in BookConference contributionpeer-review

Abstract

We present a novel framework for explainable labeling and interpretation of medical images. Medical images require specialized professionals for interpretation, and are explained (typically) via elaborate textual reports. Different from prior methods that focus on medical report generation from images or vice-versa, we novelly generate congruent image–report pairs employing a cyclic-Generative Adversarial Network (cycleGAN); thereby, the generated report will adequately explain a medical image, while a report-generated image that effectively characterizes the text visually should (sufficiently) resemble the original. The aim of the work is to generate trustworthy and faithful explanations for the outputs of a model diagnosing chest X-ray images by pointing a human user to similar cases in support of a diagnostic decision. Apart from enabling transparent medical image labeling and interpretation, we achieve report and image-based labeling comparable to prior methods, including state-of-the-art performance in some cases as evidenced by experiments on the Indiana Chest X-ray dataset.

Original languageEnglish
Title of host publicationInterpretability of Machine Intelligence in Medical Image Computing, and Topological Data Analysis and Its Applications for Medical Data - 4th International Workshop, iMIMIC 2021, and 1st International Workshop, TDA4MedicalData 2021, Held in Conjunction with MICCAI 2021, Proceedings
EditorsMauricio Reyes, Pedro Henriques Abreu, Jaime Cardoso, Mustafa Hajij, Ghada Zamzmi, Paul Rahul, Lokendra Thakur
Place of PublicationNetherlands
PublisherSpringer
Pages34-43
Number of pages10
ISBN (Print)9783030874438
DOIs
Publication statusPublished - 2021
Event4th International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2020 and 1st International Workshop on Topological Data Analysis and Its Applications for Medical Data, TDA4MedicalData 2021 held in conjunction with 24th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2021 - Strasbourg, France
Duration: 27 Sep 202127 Sep 2021

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12929 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference4th International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2020 and 1st International Workshop on Topological Data Analysis and Its Applications for Medical Data, TDA4MedicalData 2021 held in conjunction with 24th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2021
Country/TerritoryFrance
CityStrasbourg
Period27/09/2127/09/21

Fingerprint

Dive into the research topics of 'This Explains That: Congruent Image–Report Generation for Explainable Medical Image Analysis with Cyclic Generative Adversarial Networks'. Together they form a unique fingerprint.

Cite this