Efficient Labelling of Affective Video Datasets via Few-Shot & Multi-Task Contrastive Learning

Ravikiran Parameshwara, Ibrahim Radwan, Akshay Asthana, Iman Abbasnejad, Ramanathan Subramanian, Roland Goecke

Research output: A Conference proceeding or a Chapter in BookConference contributionpeer-review

1 Citation (Scopus)

Abstract

Whilst deep learning techniques have achieved excellent emotion prediction, they still require large amounts of labelled training data, which are (a) onerous and tedious to compile, and (b) prone to errors and biases. We propose Multi-Task Contrastive Learning for Affect Representation (MT-CLAR) for few-shot affect inference. MT-CLAR combines multi-task learning with a Siamese network trained via contrastive learning to infer from a pair of expressive facial images (a) the (dis)similarity between the facial expressions, and (b) the difference in valence and arousal levels of the two faces. We further extend the image-based MT-CLAR framework for automated video labelling where, given one or a few labelled video frames (termed support-set), MT-CLAR labels the remainder of the video for valence and arousal. Experiments are performed on the AFEW-VA dataset with multiple support-set configurations; moreover, supervised learning on representations learnt via MT-CLAR are used for valence, arousal and categorical emotion prediction on the AffectNet and AFEW-VA datasets. The results show that valence and arousal predictions via MT-CLAR are very comparable to the state-of-the-art (SOTA), and we significantly outperform SOTA with a support-set ~6% the size of the video dataset.
Original languageEnglish
Title of host publicationMM 2023 - Proceedings of the 31st ACM International Conference on Multimedia
EditorsMarco Bertini, Diana Patricia Tobon Vallejo, Pradeep K. Atrey, M. Shamim Hossain
Place of PublicationUnited States
PublisherAssociation for Computing Machinery (ACM)
Pages6161-6170
Number of pages10
ISBN (Electronic)9798400701085
ISBN (Print)9798400701085
DOIs
Publication statusPublished - 26 Oct 2023
Event31st ACM International Conference on Multimedia 2023 - Westin Ottawa Hotel, Ottawa, Canada
Duration: 29 Oct 20233 Nov 2023
https://www.acmmm2023.org/

Publication series

NameMM 2023 - Proceedings of the 31st ACM International Conference on Multimedia

Conference

Conference31st ACM International Conference on Multimedia 2023
Abbreviated titleACMMM '23
Country/TerritoryCanada
CityOttawa
Period29/10/233/11/23
Internet address

Fingerprint

Dive into the research topics of 'Efficient Labelling of Affective Video Datasets via Few-Shot & Multi-Task Contrastive Learning'. Together they form a unique fingerprint.

Cite this