Diagnosis of depression by behavioural signals: A multimodal approach

Nicholas Cummins, Jyoti Joshi, Abhinav Dhall, Vidhyasaharan Sethu, Roland Goecke, Julien Epps

Research output: A Conference proceeding or a Chapter in BookConference contribution

47 Citations (Scopus)

Abstract

Quantifying behavioural changes in depression using affective computing techniques is the first step in developing an objective diagnostic aid, with clinical utility, for clinical depression. As part of the AVEC 2013 Challenge, we present a multimodal approach for the Depression Sub-Challenge using a GMM-UBM system with three different kernels for the audio subsystem and Space Time Interest Points in a Bag-of-Words approach for the vision subsystem. These are then fused at the feature level to form the combined AV system. Key results include the strong performance of acoustic audio features and the bag-of-words visual features in predicting an individual’s level of depression using regression. Interestingly, in the context of the small amount of literature on the subject, is that our feature level multimodal fusion technique is able to outperform both the audio and visual challenge baselines.
Original languageEnglish
Title of host publicationAVEC 2013 - Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge
EditorsBjorn Schuller, Michel Valstar, Roddy Cowie, Maja Pantic, Jarek Krajewski
Place of PublicationBarcelona, Spain
PublisherAssociation for Computing Machinery (ACM)
Pages11-20
Number of pages10
ISBN (Print)9781450323956
DOIs
Publication statusPublished - 2013
EventACM International Workshop on Audio/Visual Emotion Challenge - Barcelona, Barcelona, Spain
Duration: 21 Oct 201325 Oct 2013

Conference

ConferenceACM International Workshop on Audio/Visual Emotion Challenge
CountrySpain
CityBarcelona
Period21/10/1325/10/13

Fingerprint

Audio acoustics
Fusion reactions

Cite this

Cummins, N., Joshi, J., Dhall, A., Sethu, V., Goecke, R., & Epps, J. (2013). Diagnosis of depression by behavioural signals: A multimodal approach. In B. Schuller, M. Valstar, R. Cowie, M. Pantic, & J. Krajewski (Eds.), AVEC 2013 - Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge (pp. 11-20). Barcelona, Spain: Association for Computing Machinery (ACM). https://doi.org/10.1145/2512530.2512535
Cummins, Nicholas ; Joshi, Jyoti ; Dhall, Abhinav ; Sethu, Vidhyasaharan ; Goecke, Roland ; Epps, Julien. / Diagnosis of depression by behavioural signals: A multimodal approach. AVEC 2013 - Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge. editor / Bjorn Schuller ; Michel Valstar ; Roddy Cowie ; Maja Pantic ; Jarek Krajewski. Barcelona, Spain : Association for Computing Machinery (ACM), 2013. pp. 11-20
@inproceedings{40b187413478454c8ccb8419e21d389f,
title = "Diagnosis of depression by behavioural signals: A multimodal approach",
abstract = "Quantifying behavioural changes in depression using affective computing techniques is the first step in developing an objective diagnostic aid, with clinical utility, for clinical depression. As part of the AVEC 2013 Challenge, we present a multimodal approach for the Depression Sub-Challenge using a GMM-UBM system with three different kernels for the audio subsystem and Space Time Interest Points in a Bag-of-Words approach for the vision subsystem. These are then fused at the feature level to form the combined AV system. Key results include the strong performance of acoustic audio features and the bag-of-words visual features in predicting an individual’s level of depression using regression. Interestingly, in the context of the small amount of literature on the subject, is that our feature level multimodal fusion technique is able to outperform both the audio and visual challenge baselines.",
keywords = "Depression, Behavioural Signals, Multimodal Technologies",
author = "Nicholas Cummins and Jyoti Joshi and Abhinav Dhall and Vidhyasaharan Sethu and Roland Goecke and Julien Epps",
year = "2013",
doi = "10.1145/2512530.2512535",
language = "English",
isbn = "9781450323956",
pages = "11--20",
editor = "Bjorn Schuller and Michel Valstar and Roddy Cowie and Maja Pantic and Jarek Krajewski",
booktitle = "AVEC 2013 - Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge",
publisher = "Association for Computing Machinery (ACM)",
address = "United States",

}

Cummins, N, Joshi, J, Dhall, A, Sethu, V, Goecke, R & Epps, J 2013, Diagnosis of depression by behavioural signals: A multimodal approach. in B Schuller, M Valstar, R Cowie, M Pantic & J Krajewski (eds), AVEC 2013 - Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge. Association for Computing Machinery (ACM), Barcelona, Spain, pp. 11-20, ACM International Workshop on Audio/Visual Emotion Challenge, Barcelona, Spain, 21/10/13. https://doi.org/10.1145/2512530.2512535

Diagnosis of depression by behavioural signals: A multimodal approach. / Cummins, Nicholas; Joshi, Jyoti; Dhall, Abhinav; Sethu, Vidhyasaharan; Goecke, Roland; Epps, Julien.

AVEC 2013 - Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge. ed. / Bjorn Schuller; Michel Valstar; Roddy Cowie; Maja Pantic; Jarek Krajewski. Barcelona, Spain : Association for Computing Machinery (ACM), 2013. p. 11-20.

Research output: A Conference proceeding or a Chapter in BookConference contribution

TY - GEN

T1 - Diagnosis of depression by behavioural signals: A multimodal approach

AU - Cummins, Nicholas

AU - Joshi, Jyoti

AU - Dhall, Abhinav

AU - Sethu, Vidhyasaharan

AU - Goecke, Roland

AU - Epps, Julien

PY - 2013

Y1 - 2013

N2 - Quantifying behavioural changes in depression using affective computing techniques is the first step in developing an objective diagnostic aid, with clinical utility, for clinical depression. As part of the AVEC 2013 Challenge, we present a multimodal approach for the Depression Sub-Challenge using a GMM-UBM system with three different kernels for the audio subsystem and Space Time Interest Points in a Bag-of-Words approach for the vision subsystem. These are then fused at the feature level to form the combined AV system. Key results include the strong performance of acoustic audio features and the bag-of-words visual features in predicting an individual’s level of depression using regression. Interestingly, in the context of the small amount of literature on the subject, is that our feature level multimodal fusion technique is able to outperform both the audio and visual challenge baselines.

AB - Quantifying behavioural changes in depression using affective computing techniques is the first step in developing an objective diagnostic aid, with clinical utility, for clinical depression. As part of the AVEC 2013 Challenge, we present a multimodal approach for the Depression Sub-Challenge using a GMM-UBM system with three different kernels for the audio subsystem and Space Time Interest Points in a Bag-of-Words approach for the vision subsystem. These are then fused at the feature level to form the combined AV system. Key results include the strong performance of acoustic audio features and the bag-of-words visual features in predicting an individual’s level of depression using regression. Interestingly, in the context of the small amount of literature on the subject, is that our feature level multimodal fusion technique is able to outperform both the audio and visual challenge baselines.

KW - Depression

KW - Behavioural Signals

KW - Multimodal Technologies

U2 - 10.1145/2512530.2512535

DO - 10.1145/2512530.2512535

M3 - Conference contribution

SN - 9781450323956

SP - 11

EP - 20

BT - AVEC 2013 - Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge

A2 - Schuller, Bjorn

A2 - Valstar, Michel

A2 - Cowie, Roddy

A2 - Pantic, Maja

A2 - Krajewski, Jarek

PB - Association for Computing Machinery (ACM)

CY - Barcelona, Spain

ER -

Cummins N, Joshi J, Dhall A, Sethu V, Goecke R, Epps J. Diagnosis of depression by behavioural signals: A multimodal approach. In Schuller B, Valstar M, Cowie R, Pantic M, Krajewski J, editors, AVEC 2013 - Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge. Barcelona, Spain: Association for Computing Machinery (ACM). 2013. p. 11-20 https://doi.org/10.1145/2512530.2512535