MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings

Surbhi Madan; Rishabh Jain; Gulshan Sharma; Ramanathan Subramanian; Abhinav Dhall

doi:10.1145/3581783.3612858

MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings

Surbhi Madan, Rishabh Jain, Gulshan Sharma, Ramanathan Subramanian, Abhinav Dhall

Information Systems

Research output: A Conference proceeding or a Chapter in Book › Conference contribution › peer-review

1 Citation (Scopus)

Abstract

Bodily behavioral language is an important social cue, and its automated analysis helps in enhancing the understanding of artificial intelligence systems. Furthermore, behavioral language cues are essential for active engagement in social agent-based user interactions. Despite the progress made in computer vision for tasks like head and body pose estimation, there is still a need to explore the detection of finer behaviors such as gesturing, grooming, or fumbling. This paper proposes a multiview attention fusion method named MAGIC-TBR that combines features extracted from videos and their corresponding Discrete Cosine Transform coefficients via a transformer-based approach. The experiments are conducted on the BBSI dataset and the results demonstrate the effectiveness of the proposed feature fusion with multiview attention. The code is available at: https://github.com/surbhimadan92/MAGIC-TBR

Original language	English
Title of host publication	Proceedings of the 31st ACM International Conference on Multimedia
Subtitle of host publication	MM23
Editors	Abdulmotaleb El Saddik, Tao Mei , Rita Cucchiara, Marco Bertini, Diana Patricia, Pradeep K. Atrey, M. Shamim Hossain, Mukesh K. Saini, Ming-Ching Chang
Place of Publication	United States
Publisher	Association for Computing Machinery (ACM)
Pages	9526-9530
Number of pages	5
ISBN (Electronic)	9798400701085
ISBN (Print)	9798400701085
DOIs	https://doi.org/10.1145/3581783.3612858
Publication status	Published - 26 Oct 2023
Event	31st ACM International Conference on Multimedia 2023 - Westin Ottawa Hotel, Ottawa, Canada Duration: 29 Oct 2023 → 3 Nov 2023 https://www.acmmm2023.org/

Publication series

Name	MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia

Conference

Conference	31st ACM International Conference on Multimedia 2023
Abbreviated title	ACMMM '23
Country/Territory	Canada
City	Ottawa
Period	29/10/23 → 3/11/23
Internet address	https://www.acmmm2023.org/

Access to Document

10.1145/3581783.3612858Licence: Other

Cite this

Madan, S., Jain, R., Sharma, G., Subramanian, R., & Dhall, A. (2023). MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings. In A. El Saddik, T. Mei , R. Cucchiara, M. Bertini, D. Patricia, P. K. Atrey, M. S. Hossain, M. K. Saini, & M.-C. Chang (Eds.), Proceedings of the 31st ACM International Conference on Multimedia: MM23 (pp. 9526-9530). (MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia). Association for Computing Machinery (ACM). https://doi.org/10.1145/3581783.3612858

Madan, Surbhi ; Jain, Rishabh ; Sharma, Gulshan et al. / MAGIC-TBR : Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings. Proceedings of the 31st ACM International Conference on Multimedia: MM23. editor / Abdulmotaleb El Saddik ; Tao Mei ; Rita Cucchiara ; Marco Bertini ; Diana Patricia ; Pradeep K. Atrey ; M. Shamim Hossain ; Mukesh K. Saini ; Ming-Ching Chang. United States : Association for Computing Machinery (ACM), 2023. pp. 9526-9530 (MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia).

@inproceedings{fec62c68647b43479c59d3681f2efe5e,

title = "MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings",

abstract = "Bodily behavioral language is an important social cue, and its automated analysis helps in enhancing the understanding of artificial intelligence systems. Furthermore, behavioral language cues are essential for active engagement in social agent-based user interactions. Despite the progress made in computer vision for tasks like head and body pose estimation, there is still a need to explore the detection of finer behaviors such as gesturing, grooming, or fumbling. This paper proposes a multiview attention fusion method named MAGIC-TBR that combines features extracted from videos and their corresponding Discrete Cosine Transform coefficients via a transformer-based approach. The experiments are conducted on the BBSI dataset and the results demonstrate the effectiveness of the proposed feature fusion with multiview attention. The code is available at: https://github.com/surbhimadan92/MAGIC-TBR",

keywords = "bodily behavior, dct, multiview attention, transformer",

author = "Surbhi Madan and Rishabh Jain and Gulshan Sharma and Ramanathan Subramanian and Abhinav Dhall",

note = "Publisher Copyright: {\textcopyright} 2023 ACM.; 31st ACM International Conference on Multimedia 2023, ACMMM '23 ; Conference date: 29-10-2023 Through 03-11-2023",

year = "2023",

month = oct,

day = "26",

doi = "10.1145/3581783.3612858",

language = "English",

isbn = "9798400701085",

series = "MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia",

publisher = "Association for Computing Machinery (ACM)",

pages = "9526--9530",

editor = "{ El Saddik}, Abdulmotaleb and {Mei }, {Tao } and Cucchiara, {Rita } and Bertini, {Marco } and Patricia, {Diana } and {K. Atrey}, {Pradeep } and Hossain, {M. Shamim } and { K. Saini}, Mukesh and Chang, {Ming-Ching }",

booktitle = "Proceedings of the 31st ACM International Conference on Multimedia",

address = "United States",

url = "https://www.acmmm2023.org/",

}

Madan, S, Jain, R, Sharma, G, Subramanian, R & Dhall, A 2023, MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings. in A El Saddik, T Mei , R Cucchiara, M Bertini, D Patricia, P K. Atrey, MS Hossain, M K. Saini & M-C Chang (eds), Proceedings of the 31st ACM International Conference on Multimedia: MM23. MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia, Association for Computing Machinery (ACM), United States, pp. 9526-9530, 31st ACM International Conference on Multimedia 2023, Ottawa, Ontario, Canada, 29/10/23. https://doi.org/10.1145/3581783.3612858

MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings. / Madan, Surbhi; Jain, Rishabh; Sharma, Gulshan et al.
Proceedings of the 31st ACM International Conference on Multimedia: MM23. ed. / Abdulmotaleb El Saddik; Tao Mei ; Rita Cucchiara; Marco Bertini; Diana Patricia; Pradeep K. Atrey; M. Shamim Hossain; Mukesh K. Saini; Ming-Ching Chang. United States: Association for Computing Machinery (ACM), 2023. p. 9526-9530 (MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia).

Research output: A Conference proceeding or a Chapter in Book › Conference contribution › peer-review

TY - GEN

T1 - MAGIC-TBR

T2 - 31st ACM International Conference on Multimedia 2023

AU - Madan, Surbhi

AU - Jain, Rishabh

AU - Sharma, Gulshan

AU - Subramanian, Ramanathan

AU - Dhall, Abhinav

PY - 2023/10/26

Y1 - 2023/10/26

N2 - Bodily behavioral language is an important social cue, and its automated analysis helps in enhancing the understanding of artificial intelligence systems. Furthermore, behavioral language cues are essential for active engagement in social agent-based user interactions. Despite the progress made in computer vision for tasks like head and body pose estimation, there is still a need to explore the detection of finer behaviors such as gesturing, grooming, or fumbling. This paper proposes a multiview attention fusion method named MAGIC-TBR that combines features extracted from videos and their corresponding Discrete Cosine Transform coefficients via a transformer-based approach. The experiments are conducted on the BBSI dataset and the results demonstrate the effectiveness of the proposed feature fusion with multiview attention. The code is available at: https://github.com/surbhimadan92/MAGIC-TBR

AB - Bodily behavioral language is an important social cue, and its automated analysis helps in enhancing the understanding of artificial intelligence systems. Furthermore, behavioral language cues are essential for active engagement in social agent-based user interactions. Despite the progress made in computer vision for tasks like head and body pose estimation, there is still a need to explore the detection of finer behaviors such as gesturing, grooming, or fumbling. This paper proposes a multiview attention fusion method named MAGIC-TBR that combines features extracted from videos and their corresponding Discrete Cosine Transform coefficients via a transformer-based approach. The experiments are conducted on the BBSI dataset and the results demonstrate the effectiveness of the proposed feature fusion with multiview attention. The code is available at: https://github.com/surbhimadan92/MAGIC-TBR

KW - bodily behavior

KW - dct

KW - multiview attention

KW - transformer

UR - https://www.acmmm2023.org/

UR - http://www.scopus.com/inward/record.url?scp=85179553773&partnerID=8YFLogxK

U2 - 10.1145/3581783.3612858

DO - 10.1145/3581783.3612858

M3 - Conference contribution

SN - 9798400701085

T3 - MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia

SP - 9526

EP - 9530

BT - Proceedings of the 31st ACM International Conference on Multimedia

A2 - El Saddik, Abdulmotaleb

A2 - Mei , Tao

A2 - Cucchiara, Rita

A2 - Bertini, Marco

A2 - Patricia, Diana

A2 - K. Atrey, Pradeep

A2 - Hossain, M. Shamim

A2 - K. Saini, Mukesh

A2 - Chang, Ming-Ching

PB - Association for Computing Machinery (ACM)

CY - United States

Y2 - 29 October 2023 through 3 November 2023

ER -

Madan S, Jain R, Sharma G, Subramanian R, Dhall A. MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings. In El Saddik A, Mei T, Cucchiara R, Bertini M, Patricia D, K. Atrey P, Hossain MS, K. Saini M, Chang MC, editors, Proceedings of the 31st ACM International Conference on Multimedia: MM23. United States: Association for Computing Machinery (ACM). 2023. p. 9526-9530. (MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia). doi: 10.1145/3581783.3612858

MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this