A multimodal system to characterise melancholia: Cascaded bag of words approach

Shalini Bhatia, Munawar Hayat, Roland Goecke

Research output: A Conference proceeding or a Chapter in BookConference contribution

2 Citations (Scopus)

Abstract

Recent years have seen a lot of activity in affective computing for automated analysis of depression. However, no research has so far proposed a multimodal system for classifying different subtypes of depression such as melancholia. The mental state assessment of a mood disorder depends primarily on appearance, behaviour, speech, thought, perception, mood and facial affect. Mood and facial affect mainly contribute to distinguishing melancholia from nonmelancholia. These are assessed by clinicians, and hence vulnerable to subjective judgement. As a result, clinical assessment alone may not accurately capture the presence or absence of specific disorders such as melancholia, a distressing condition whose presence has important treatment implications. Melancholia is characterised by severe anhedonia and psychomotor disturbance, which can be a combination of motor retardation with periods of superimposed agitation. Psychomotor disturbance can be sensed in both face and voice. To the best of our knowledge, this study is the first attempt to propose a multimodal system to differentiate melancholia from non-melancholia and healthy controls. We report the sensitivity and specificity of classification in depressive subtypes.

Original languageEnglish
Title of host publicationICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction
Place of PublicationGlasgow, United Kingdom
PublisherAssociation for Computing Machinery, Inc
Pages274-280
Number of pages7
ISBN (Electronic)9781450355438
ISBN (Print)9781450355438
DOIs
Publication statusPublished - 3 Nov 2017
Event19th ACM International Conference on Multimodal Interaction, ICMI 2017 - Glasgow, United Kingdom
Duration: 13 Nov 201717 Nov 2017

Publication series

NameICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction
Volume2017-January

Conference

Conference19th ACM International Conference on Multimodal Interaction, ICMI 2017
CountryUnited Kingdom
CityGlasgow
Period13/11/1717/11/17

Cite this

Bhatia, S., Hayat, M., & Goecke, R. (2017). A multimodal system to characterise melancholia: Cascaded bag of words approach. In ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction (pp. 274-280). (ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction; Vol. 2017-January). Glasgow, United Kingdom: Association for Computing Machinery, Inc. https://doi.org/10.1145/3136755.3136766
Bhatia, Shalini ; Hayat, Munawar ; Goecke, Roland. / A multimodal system to characterise melancholia: Cascaded bag of words approach. ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction. Glasgow, United Kingdom : Association for Computing Machinery, Inc, 2017. pp. 274-280 (ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction).
@inproceedings{1bd1d4021d374ee896114b5cd8b94041,
title = "A multimodal system to characterise melancholia: Cascaded bag of words approach",
abstract = "Recent years have seen a lot of activity in affective computing for automated analysis of depression. However, no research has so far proposed a multimodal system for classifying different subtypes of depression such as melancholia. The mental state assessment of a mood disorder depends primarily on appearance, behaviour, speech, thought, perception, mood and facial affect. Mood and facial affect mainly contribute to distinguishing melancholia from nonmelancholia. These are assessed by clinicians, and hence vulnerable to subjective judgement. As a result, clinical assessment alone may not accurately capture the presence or absence of specific disorders such as melancholia, a distressing condition whose presence has important treatment implications. Melancholia is characterised by severe anhedonia and psychomotor disturbance, which can be a combination of motor retardation with periods of superimposed agitation. Psychomotor disturbance can be sensed in both face and voice. To the best of our knowledge, this study is the first attempt to propose a multimodal system to differentiate melancholia from non-melancholia and healthy controls. We report the sensitivity and specificity of classification in depressive subtypes.",
keywords = "Audio-Video, Bag of words, Multimodal fusion, Melancholia, Depression analysis, Multimodal, Video, Audio, Fusion",
author = "Shalini Bhatia and Munawar Hayat and Roland Goecke",
year = "2017",
month = "11",
day = "3",
doi = "10.1145/3136755.3136766",
language = "English",
isbn = "9781450355438",
series = "ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction",
publisher = "Association for Computing Machinery, Inc",
pages = "274--280",
booktitle = "ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction",

}

Bhatia, S, Hayat, M & Goecke, R 2017, A multimodal system to characterise melancholia: Cascaded bag of words approach. in ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction. ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction, vol. 2017-January, Association for Computing Machinery, Inc, Glasgow, United Kingdom, pp. 274-280, 19th ACM International Conference on Multimodal Interaction, ICMI 2017, Glasgow, United Kingdom, 13/11/17. https://doi.org/10.1145/3136755.3136766

A multimodal system to characterise melancholia: Cascaded bag of words approach. / Bhatia, Shalini; Hayat, Munawar; Goecke, Roland.

ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction. Glasgow, United Kingdom : Association for Computing Machinery, Inc, 2017. p. 274-280 (ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction; Vol. 2017-January).

Research output: A Conference proceeding or a Chapter in BookConference contribution

TY - GEN

T1 - A multimodal system to characterise melancholia: Cascaded bag of words approach

AU - Bhatia, Shalini

AU - Hayat, Munawar

AU - Goecke, Roland

PY - 2017/11/3

Y1 - 2017/11/3

N2 - Recent years have seen a lot of activity in affective computing for automated analysis of depression. However, no research has so far proposed a multimodal system for classifying different subtypes of depression such as melancholia. The mental state assessment of a mood disorder depends primarily on appearance, behaviour, speech, thought, perception, mood and facial affect. Mood and facial affect mainly contribute to distinguishing melancholia from nonmelancholia. These are assessed by clinicians, and hence vulnerable to subjective judgement. As a result, clinical assessment alone may not accurately capture the presence or absence of specific disorders such as melancholia, a distressing condition whose presence has important treatment implications. Melancholia is characterised by severe anhedonia and psychomotor disturbance, which can be a combination of motor retardation with periods of superimposed agitation. Psychomotor disturbance can be sensed in both face and voice. To the best of our knowledge, this study is the first attempt to propose a multimodal system to differentiate melancholia from non-melancholia and healthy controls. We report the sensitivity and specificity of classification in depressive subtypes.

AB - Recent years have seen a lot of activity in affective computing for automated analysis of depression. However, no research has so far proposed a multimodal system for classifying different subtypes of depression such as melancholia. The mental state assessment of a mood disorder depends primarily on appearance, behaviour, speech, thought, perception, mood and facial affect. Mood and facial affect mainly contribute to distinguishing melancholia from nonmelancholia. These are assessed by clinicians, and hence vulnerable to subjective judgement. As a result, clinical assessment alone may not accurately capture the presence or absence of specific disorders such as melancholia, a distressing condition whose presence has important treatment implications. Melancholia is characterised by severe anhedonia and psychomotor disturbance, which can be a combination of motor retardation with periods of superimposed agitation. Psychomotor disturbance can be sensed in both face and voice. To the best of our knowledge, this study is the first attempt to propose a multimodal system to differentiate melancholia from non-melancholia and healthy controls. We report the sensitivity and specificity of classification in depressive subtypes.

KW - Audio-Video

KW - Bag of words

KW - Multimodal fusion

KW - Melancholia

KW - Depression analysis

KW - Multimodal

KW - Video

KW - Audio

KW - Fusion

UR - http://www.scopus.com/inward/record.url?scp=85046636068&partnerID=8YFLogxK

UR - https://dl.acm.org/citation.cfm?doid=3136755.3136766

UR - http://www.mendeley.com/research/multimodal-system-characterise-melancholia-cascaded-bag-words-approach

U2 - 10.1145/3136755.3136766

DO - 10.1145/3136755.3136766

M3 - Conference contribution

SN - 9781450355438

T3 - ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction

SP - 274

EP - 280

BT - ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction

PB - Association for Computing Machinery, Inc

CY - Glasgow, United Kingdom

ER -

Bhatia S, Hayat M, Goecke R. A multimodal system to characterise melancholia: Cascaded bag of words approach. In ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction. Glasgow, United Kingdom: Association for Computing Machinery, Inc. 2017. p. 274-280. (ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction). https://doi.org/10.1145/3136755.3136766