MIP-GAF: A MLLM-annotated Benchmark for Important Person Localization and Group Context Understanding

Surbhi Madan; Shreya Ghosh; Lownish Rai Sookha; M.A. Ganaie; Ramanathan Subramanian; Abhinav  Dhall; Tom Gedeon

doi:10.48550/arXiv.2409.06224

MIP-GAF: A MLLM-annotated Benchmark for Important Person Localization and Group Context Understanding

Surbhi Madan, Shreya Ghosh, Lownish Rai Sookha, M.A. Ganaie, Ramanathan Subramanian, Abhinav Dhall, Tom Gedeon

AI and Robotics

Research output: A Conference proceeding or a Chapter in Book › Conference contribution › peer-review

Abstract

Estimating the Most Important Person (MIP) in any social event setup is a challenging problem mainly due to contextual complexity and scarcity of labeled data. Moreover, the causality aspects of MIP estimation are quite subjective and diverse. To this end, we aim to address the problem by annotating a large-scale `in-the-wild' dataset for identifying human perceptions about the `Most Important Person (MIP)' in an image. The paper provides a thorough description of our proposed Multimodal Large Language Model (MLLM) based data annotation strategy, and a thorough data quality analysis. Further, we perform a comprehensive benchmarking of the proposed dataset utilizing state-of-the-art MIP localization methods, indicating a significant drop in performance compared to existing datasets. The performance drop shows that the existing MIP localization algorithms must be more robust with respect to `in-the-wild' situations. We believe the proposed dataset will play a vital role in building the next-generation social situation understanding methods. The code and data is available at https://github.com/surbhimadan92/MIP-GAF

Original language	English
Title of host publication	IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Editors	Sharon X. Huang, Peyman Milanfar, Vishal M. Patel, Soma Biswas, Hadar Averbuch-Elor, Vitomir Štruc, Yezhou Yang
Place of Publication	United States
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Pages	1-10
Number of pages	10
DOIs	https://doi.org/10.48550/arXiv.2409.06224
Publication status	Published - 2025
Event	IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) - Tucson, United States Duration: 28 Feb 2025 → 4 Mar 2025 https://wacv2025.thecvf.com/

Publication series

Name	Winter Conference on Applications of Computer Vision (WACV)
Publisher	IEEE
ISSN (Print)	2379-8920
ISSN (Electronic)	2379-8939

Conference

Conference	IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Abbreviated title	WACV2025
Country/Territory	United States
City	Tucson
Period	28/02/25 → 4/03/25
Internet address	https://wacv2025.thecvf.com/

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.48550/arXiv.2409.06224

Cite this

Madan, S., Ghosh, S., Sookha, L. R., Ganaie, M. A., Subramanian, R., Dhall, A., & Gedeon, T. (2025). MIP-GAF: A MLLM-annotated Benchmark for Important Person Localization and Group Context Understanding. In S. X. Huang, P. Milanfar, V. M. Patel, S. Biswas, H. Averbuch-Elor, V. Štruc, & Y. Yang (Eds.), IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (pp. 1-10). (Winter Conference on Applications of Computer Vision (WACV)). IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.48550/arXiv.2409.06224

Madan, Surbhi ; Ghosh, Shreya ; Sookha, Lownish Rai et al. / MIP-GAF: A MLLM-annotated Benchmark for Important Person Localization and Group Context Understanding. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). editor / Sharon X. Huang ; Peyman Milanfar ; Vishal M. Patel ; Soma Biswas ; Hadar Averbuch-Elor ; Vitomir Štruc ; Yezhou Yang. United States : IEEE, Institute of Electrical and Electronics Engineers, 2025. pp. 1-10 (Winter Conference on Applications of Computer Vision (WACV)).

@inproceedings{5f9f15e36daf4183803fa8563405b9de,

title = "MIP-GAF: A MLLM-annotated Benchmark for Important Person Localization and Group Context Understanding",

abstract = "Estimating the Most Important Person (MIP) in any social event setup is a challenging problem mainly due to contextual complexity and scarcity of labeled data. Moreover, the causality aspects of MIP estimation are quite subjective and diverse. To this end, we aim to address the problem by annotating a large-scale `in-the-wild' dataset for identifying human perceptions about the `Most Important Person (MIP)' in an image. The paper provides a thorough description of our proposed Multimodal Large Language Model (MLLM) based data annotation strategy, and a thorough data quality analysis. Further, we perform a comprehensive benchmarking of the proposed dataset utilizing state-of-the-art MIP localization methods, indicating a significant drop in performance compared to existing datasets. The performance drop shows that the existing MIP localization algorithms must be more robust with respect to `in-the-wild' situations. We believe the proposed dataset will play a vital role in building the next-generation social situation understanding methods. The code and data is available at https://github.com/surbhimadan92/MIP-GAF",

keywords = "Multimodal, Large Language Models, Scene Understanding, Most Important Person",

author = "Surbhi Madan and Shreya Ghosh and Sookha, {Lownish Rai} and M.A. Ganaie and Ramanathan Subramanian and Abhinav Dhall and Tom Gedeon",

year = "2025",

doi = "10.48550/arXiv.2409.06224",

language = "English",

series = "Winter Conference on Applications of Computer Vision (WACV)",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

pages = "1--10",

editor = "{X. Huang}, Sharon and Milanfar, {Peyman } and {M. Patel}, {Vishal } and Soma Biswas and Averbuch-Elor, {Hadar } and {\v S}truc, {Vitomir } and Yezhou Yang",

booktitle = "IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)",

address = "United States",

note = "IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , WACV2025 ; Conference date: 28-02-2025 Through 04-03-2025",

url = "https://wacv2025.thecvf.com/",

}

Madan, S, Ghosh, S, Sookha, LR, Ganaie, MA, Subramanian, R, Dhall, A & Gedeon, T 2025, MIP-GAF: A MLLM-annotated Benchmark for Important Person Localization and Group Context Understanding. in S X. Huang, P Milanfar, V M. Patel, S Biswas, H Averbuch-Elor, V Štruc & Y Yang (eds), IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Winter Conference on Applications of Computer Vision (WACV), IEEE, Institute of Electrical and Electronics Engineers, United States, pp. 1-10, IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , Tucson, United States, 28/02/25. https://doi.org/10.48550/arXiv.2409.06224

MIP-GAF: A MLLM-annotated Benchmark for Important Person Localization and Group Context Understanding. / Madan, Surbhi; Ghosh, Shreya; Sookha, Lownish Rai et al.
IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). ed. / Sharon X. Huang; Peyman Milanfar; Vishal M. Patel; Soma Biswas; Hadar Averbuch-Elor; Vitomir Štruc; Yezhou Yang. United States: IEEE, Institute of Electrical and Electronics Engineers, 2025. p. 1-10 (Winter Conference on Applications of Computer Vision (WACV)).

Research output: A Conference proceeding or a Chapter in Book › Conference contribution › peer-review

TY - GEN

T1 - MIP-GAF: A MLLM-annotated Benchmark for Important Person Localization and Group Context Understanding

AU - Madan, Surbhi

AU - Ghosh, Shreya

AU - Sookha, Lownish Rai

AU - Ganaie, M.A.

AU - Subramanian, Ramanathan

AU - Dhall, Abhinav

AU - Gedeon, Tom

PY - 2025

Y1 - 2025

N2 - Estimating the Most Important Person (MIP) in any social event setup is a challenging problem mainly due to contextual complexity and scarcity of labeled data. Moreover, the causality aspects of MIP estimation are quite subjective and diverse. To this end, we aim to address the problem by annotating a large-scale `in-the-wild' dataset for identifying human perceptions about the `Most Important Person (MIP)' in an image. The paper provides a thorough description of our proposed Multimodal Large Language Model (MLLM) based data annotation strategy, and a thorough data quality analysis. Further, we perform a comprehensive benchmarking of the proposed dataset utilizing state-of-the-art MIP localization methods, indicating a significant drop in performance compared to existing datasets. The performance drop shows that the existing MIP localization algorithms must be more robust with respect to `in-the-wild' situations. We believe the proposed dataset will play a vital role in building the next-generation social situation understanding methods. The code and data is available at https://github.com/surbhimadan92/MIP-GAF

AB - Estimating the Most Important Person (MIP) in any social event setup is a challenging problem mainly due to contextual complexity and scarcity of labeled data. Moreover, the causality aspects of MIP estimation are quite subjective and diverse. To this end, we aim to address the problem by annotating a large-scale `in-the-wild' dataset for identifying human perceptions about the `Most Important Person (MIP)' in an image. The paper provides a thorough description of our proposed Multimodal Large Language Model (MLLM) based data annotation strategy, and a thorough data quality analysis. Further, we perform a comprehensive benchmarking of the proposed dataset utilizing state-of-the-art MIP localization methods, indicating a significant drop in performance compared to existing datasets. The performance drop shows that the existing MIP localization algorithms must be more robust with respect to `in-the-wild' situations. We believe the proposed dataset will play a vital role in building the next-generation social situation understanding methods. The code and data is available at https://github.com/surbhimadan92/MIP-GAF

KW - Multimodal

KW - Large Language Models

KW - Scene Understanding

KW - Most Important Person

UR - https://github.com/surbhimadan92/MIP-GAF

UR - https://wacv2025.thecvf.com/

U2 - 10.48550/arXiv.2409.06224

DO - 10.48550/arXiv.2409.06224

M3 - Conference contribution

T3 - Winter Conference on Applications of Computer Vision (WACV)

SP - 1

EP - 10

BT - IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

A2 - X. Huang, Sharon

A2 - Milanfar, Peyman

A2 - M. Patel, Vishal

A2 - Biswas, Soma

A2 - Averbuch-Elor, Hadar

A2 - Štruc, Vitomir

A2 - Yang, Yezhou

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - United States

T2 - IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

Y2 - 28 February 2025 through 4 March 2025

ER -

Madan S, Ghosh S, Sookha LR, Ganaie MA, Subramanian R, Dhall A et al. MIP-GAF: A MLLM-annotated Benchmark for Important Person Localization and Group Context Understanding. In X. Huang S, Milanfar P, M. Patel V, Biswas S, Averbuch-Elor H, Štruc V, Yang Y, editors, IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). United States: IEEE, Institute of Electrical and Electronics Engineers. 2025. p. 1-10. (Winter Conference on Applications of Computer Vision (WACV)). doi: 10.48550/arXiv.2409.06224

MIP-GAF: A MLLM-annotated Benchmark for Important Person Localization and Group Context Understanding

Abstract

Publication series

Conference

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this