Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis

Subramanian Ramanathan; Harish Katti; Raymond Huang; Tat Seng Chua; Mohan Kankanhalli

doi:10.1145/1631272.1631399

Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis

Subramanian Ramanathan, Harish Katti, Raymond Huang, Tat Seng Chua, Mohan Kankanhalli

Research output: A Conference proceeding or a Chapter in Book › Conference contribution › peer-review

25 Citations (Scopus)

Abstract

We propose a novel framework to localize and label affective objects and actions in images through a combination of text, visual and gaze-based analysis. Human gaze provides useful cues to infer locations and interactions of affective objects. While concepts (labels) associated with an image can be determined from its caption, we demonstrate localization of these concepts upon learning from a statistical affect model for world concepts. The affect model is derived from non-invasively acquired fixation patterns on labeled images, and guides localization of affective objects (faces, reptiles) and actions (look, read) from fixations in unlabeled images. Experimental results obtained on a database of 500 images confirm the effectiveness and promise of the proposed approach.

Original language	English
Title of host publication	MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums
Editors	Wen Gao, Yong Rui, Alan Hanjalic, Changsheng Xu, Eckehard Steinbach, Abdulmotaleb El Saddik, Michelle Zhou
Place of Publication	United States
Publisher	Association for Computing Machinery (ACM)
Pages	729-732
Number of pages	4
ISBN (Print)	9781605586083
DOIs	https://doi.org/10.1145/1631272.1631399
Publication status	Published - 2009
Externally published	Yes
Event	17th ACM International Conference on Multimedia, MM'09, with Co-located Workshops and Symposiums - Beijing, China Duration: 19 Oct 2009 → 24 Oct 2009

Publication series

Name	MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums

Conference

Conference	17th ACM International Conference on Multimedia, MM'09, with Co-located Workshops and Symposiums
Country/Territory	China
City	Beijing
Period	19/10/09 → 24/10/09

Access to Document

10.1145/1631272.1631399

Cite this

Ramanathan, S., Katti, H., Huang, R., Chua, T. S., & Kankanhalli, M. (2009). Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis. In W. Gao, Y. Rui, A. Hanjalic, C. Xu, E. Steinbach, A. El Saddik, & M. Zhou (Eds.), MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums (pp. 729-732). (MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums). Association for Computing Machinery (ACM). https://doi.org/10.1145/1631272.1631399

Ramanathan, Subramanian ; Katti, Harish ; Huang, Raymond et al. / Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis. MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums. editor / Wen Gao ; Yong Rui ; Alan Hanjalic ; Changsheng Xu ; Eckehard Steinbach ; Abdulmotaleb El Saddik ; Michelle Zhou. United States : Association for Computing Machinery (ACM), 2009. pp. 729-732 (MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums).

@inproceedings{8129549dcf1842bb82453664025542fd,

title = "Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis",

abstract = "We propose a novel framework to localize and label affective objects and actions in images through a combination of text, visual and gaze-based analysis. Human gaze provides useful cues to infer locations and interactions of affective objects. While concepts (labels) associated with an image can be determined from its caption, we demonstrate localization of these concepts upon learning from a statistical affect model for world concepts. The affect model is derived from non-invasively acquired fixation patterns on labeled images, and guides localization of affective objects (faces, reptiles) and actions (look, read) from fixations in unlabeled images. Experimental results obtained on a database of 500 images confirm the effectiveness and promise of the proposed approach.",

keywords = "Affect model for world concepts, Automated localization and labeling, Caption text-cum-eye gaze analysis, Statistical model",

author = "Subramanian Ramanathan and Harish Katti and Raymond Huang and Chua, {Tat Seng} and Mohan Kankanhalli",

year = "2009",

doi = "10.1145/1631272.1631399",

language = "English",

isbn = "9781605586083",

series = "MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums",

publisher = "Association for Computing Machinery (ACM)",

pages = "729--732",

editor = "Wen Gao and Yong Rui and Alan Hanjalic and Changsheng Xu and Eckehard Steinbach and {El Saddik}, Abdulmotaleb and Michelle Zhou",

booktitle = "MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums",

address = "United States",

note = "17th ACM International Conference on Multimedia, MM'09, with Co-located Workshops and Symposiums ; Conference date: 19-10-2009 Through 24-10-2009",

}

Ramanathan, S, Katti, H, Huang, R, Chua, TS & Kankanhalli, M 2009, Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis. in W Gao, Y Rui, A Hanjalic, C Xu, E Steinbach, A El Saddik & M Zhou (eds), MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums. MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums, Association for Computing Machinery (ACM), United States, pp. 729-732, 17th ACM International Conference on Multimedia, MM'09, with Co-located Workshops and Symposiums, Beijing, China, 19/10/09. https://doi.org/10.1145/1631272.1631399

Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis. / Ramanathan, Subramanian; Katti, Harish; Huang, Raymond et al.
MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums. ed. / Wen Gao; Yong Rui; Alan Hanjalic; Changsheng Xu; Eckehard Steinbach; Abdulmotaleb El Saddik; Michelle Zhou. United States: Association for Computing Machinery (ACM), 2009. p. 729-732 (MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums).

Research output: A Conference proceeding or a Chapter in Book › Conference contribution › peer-review

TY - GEN

T1 - Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis

AU - Ramanathan, Subramanian

AU - Katti, Harish

AU - Huang, Raymond

AU - Chua, Tat Seng

AU - Kankanhalli, Mohan

PY - 2009

Y1 - 2009

N2 - We propose a novel framework to localize and label affective objects and actions in images through a combination of text, visual and gaze-based analysis. Human gaze provides useful cues to infer locations and interactions of affective objects. While concepts (labels) associated with an image can be determined from its caption, we demonstrate localization of these concepts upon learning from a statistical affect model for world concepts. The affect model is derived from non-invasively acquired fixation patterns on labeled images, and guides localization of affective objects (faces, reptiles) and actions (look, read) from fixations in unlabeled images. Experimental results obtained on a database of 500 images confirm the effectiveness and promise of the proposed approach.

AB - We propose a novel framework to localize and label affective objects and actions in images through a combination of text, visual and gaze-based analysis. Human gaze provides useful cues to infer locations and interactions of affective objects. While concepts (labels) associated with an image can be determined from its caption, we demonstrate localization of these concepts upon learning from a statistical affect model for world concepts. The affect model is derived from non-invasively acquired fixation patterns on labeled images, and guides localization of affective objects (faces, reptiles) and actions (look, read) from fixations in unlabeled images. Experimental results obtained on a database of 500 images confirm the effectiveness and promise of the proposed approach.

KW - Affect model for world concepts

KW - Automated localization and labeling

KW - Caption text-cum-eye gaze analysis

KW - Statistical model

UR - http://www.scopus.com/inward/record.url?scp=72549091135&partnerID=8YFLogxK

U2 - 10.1145/1631272.1631399

DO - 10.1145/1631272.1631399

M3 - Conference contribution

AN - SCOPUS:72549091135

SN - 9781605586083

T3 - MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums

SP - 729

EP - 732

BT - MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums

A2 - Gao, Wen

A2 - Rui, Yong

A2 - Hanjalic, Alan

A2 - Xu, Changsheng

A2 - Steinbach, Eckehard

A2 - El Saddik, Abdulmotaleb

A2 - Zhou, Michelle

PB - Association for Computing Machinery (ACM)

CY - United States

T2 - 17th ACM International Conference on Multimedia, MM'09, with Co-located Workshops and Symposiums

Y2 - 19 October 2009 through 24 October 2009

ER -

Ramanathan S, Katti H, Huang R, Chua TS, Kankanhalli M. Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis. In Gao W, Rui Y, Hanjalic A, Xu C, Steinbach E, El Saddik A, Zhou M, editors, MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums. United States: Association for Computing Machinery (ACM). 2009. p. 729-732. (MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums). doi: 10.1145/1631272.1631399

Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this