Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review

Essam Debie, Raul Fernandez Rojas, Justin Fidock, Michael Barlow, Kathryn Kasmarik, Sreenatha Anavatti, Matthew Garratt, Hussein A. Abbass

Research output: Contribution to journalReview article

Abstract

Considerable progress has been made in improving the estimation accuracy of cognitive workload using various sensor technologies. However, the overall performance of different algorithms and methods remain suboptimal in real-world applications. Some studies in the literature demonstrate that a single modality is sufficient to estimate cognitive workload. These studies are limited to controlled settings, a scenario that is significantly different from the real world where data gets corrupted, interrupted, and delayed. In such situations, the use of multiple modalities is needed. Multimodal fusion approaches have been successful in other domains, such as wireless-sensor networks, in addressing single-sensor weaknesses and improving information quality/accuracy. These approaches are inherently more reliable when a data source is lost. In the cognitive workload literature, sensors, such as electroencephalography (EEG), electrocardiography (ECG), and eye tracking, have shown success in estimating the aspects of cognitive workload. Multimodal approaches that combine data from several sensors together can be more robust for real-time measurement of cognitive workload. In this article, we review the published studies related to multimodal data fusion to estimate the cognitive workload and synthesize their main findings. We identify the opportunities for designing better multimodal fusion systems for cognitive workload modeling.
Original languageEnglish
Pages (from-to)1-14
Number of pages14
JournalIEEE Transactions on Cybernetics
DOIs
Publication statusE-pub ahead of print - 23 Sep 2019
Externally publishedYes

Fingerprint Dive into the research topics of 'Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review'. Together they form a unique fingerprint.

  • Cite this

    Debie, E., Rojas, R. F., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S., Garratt, M., & Abbass, H. A. (2019). Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review. IEEE Transactions on Cybernetics, 1-14. https://doi.org/10.1109/tcyb.2019.2939399