A Robust Spatio-temporal Face Modelling Approach Using 3D Multimodal Fusion for Biometric Security Applications

Girija Chetty, Michael Wagner

    Research output: A Conference proceeding or a Chapter in BookConference contribution

    Abstract

    In this paper, we propose a robust spatio-temporal face modelling approach based on multilevel fusion strategy involving cascaded fusion of hybrid multimodal fusion of audio-lip-face motion, correlation and depth features for biometric security application. The proposed approach combines the information from different audio-video based modules, namely: audio-lip motion module, audio-lip correlation module, 2D+3D motion-depth fusion module, and performs a hybrid cascaded fusion in an automatic, unsupervised and adaptive manner, by adapting to the local performance of each module. This is done by taking the output-score based reliability estimates (confidence measures) of each of the module into account. The module weightings are determined automatically such that the reliability measure of the combined scores is maximised. To test the robustness of the proposed approach, the audio and visual speech (mouth) modalities are degraded to emulate various levels of train/test mismatch; employing additive white Gaussian noise for the audio and JPEG compression for the video signals. The results show improved fusion performance for a range of tested levels of audio and video degradation, compared to the individual module performances. Experiments on a 3D stereovision database AVOZES show that, at severe levels of audio and video mismatch, the audio, mouth, 3D face, and tri-module (audio-lip motion, correlation and depth) fusion EERs were 42.9%, 32%, 15%, and 7.3% respectively for biometric identity verification scenario
    Original languageEnglish
    Title of host publicationProceedings of the SPIE, Biometric Technology for Human Identification V
    EditorsB.V.K Vijaya Kumar, Salil Prabhakar, Arun Ross
    Place of PublicationWashington
    PublisherSPIE
    Pages1-5
    Number of pages5
    Volume6944
    DOIs
    Publication statusPublished - 2008
    EventSPIE Conference on Biometric Technology for Human Identification V - Orlando, United States
    Duration: 18 Mar 200819 Mar 2008

    Publication series

    NameProceedings of SPIE
    PublisherSPIE
    Volume6944
    ISSN (Print)0277-786X

    Conference

    ConferenceSPIE Conference on Biometric Technology for Human Identification V
    CountryUnited States
    CityOrlando
    Period18/03/0819/03/08

    Fingerprint

    Biometrics
    Fusion reactions
    Degradation
    Experiments

    Cite this

    Chetty, G., & Wagner, M. (2008). A Robust Spatio-temporal Face Modelling Approach Using 3D Multimodal Fusion for Biometric Security Applications. In B. V. K. V. Kumar, S. Prabhakar, & A. Ross (Eds.), Proceedings of the SPIE, Biometric Technology for Human Identification V (Vol. 6944, pp. 1-5). (Proceedings of SPIE; Vol. 6944). Washington: SPIE. https://doi.org/10.1117/12.778631
    Chetty, Girija ; Wagner, Michael. / A Robust Spatio-temporal Face Modelling Approach Using 3D Multimodal Fusion for Biometric Security Applications. Proceedings of the SPIE, Biometric Technology for Human Identification V. editor / B.V.K Vijaya Kumar ; Salil Prabhakar ; Arun Ross. Vol. 6944 Washington : SPIE, 2008. pp. 1-5 (Proceedings of SPIE).
    @inproceedings{c64aa0d55bc4431bac62ee6cf20e7887,
    title = "A Robust Spatio-temporal Face Modelling Approach Using 3D Multimodal Fusion for Biometric Security Applications",
    abstract = "In this paper, we propose a robust spatio-temporal face modelling approach based on multilevel fusion strategy involving cascaded fusion of hybrid multimodal fusion of audio-lip-face motion, correlation and depth features for biometric security application. The proposed approach combines the information from different audio-video based modules, namely: audio-lip motion module, audio-lip correlation module, 2D+3D motion-depth fusion module, and performs a hybrid cascaded fusion in an automatic, unsupervised and adaptive manner, by adapting to the local performance of each module. This is done by taking the output-score based reliability estimates (confidence measures) of each of the module into account. The module weightings are determined automatically such that the reliability measure of the combined scores is maximised. To test the robustness of the proposed approach, the audio and visual speech (mouth) modalities are degraded to emulate various levels of train/test mismatch; employing additive white Gaussian noise for the audio and JPEG compression for the video signals. The results show improved fusion performance for a range of tested levels of audio and video degradation, compared to the individual module performances. Experiments on a 3D stereovision database AVOZES show that, at severe levels of audio and video mismatch, the audio, mouth, 3D face, and tri-module (audio-lip motion, correlation and depth) fusion EERs were 42.9{\%}, 32{\%}, 15{\%}, and 7.3{\%} respectively for biometric identity verification scenario",
    author = "Girija Chetty and Michael Wagner",
    year = "2008",
    doi = "10.1117/12.778631",
    language = "English",
    volume = "6944",
    series = "Proceedings of SPIE",
    publisher = "SPIE",
    pages = "1--5",
    editor = "Kumar, {B.V.K Vijaya} and Salil Prabhakar and Arun Ross",
    booktitle = "Proceedings of the SPIE, Biometric Technology for Human Identification V",
    address = "United States",

    }

    Chetty, G & Wagner, M 2008, A Robust Spatio-temporal Face Modelling Approach Using 3D Multimodal Fusion for Biometric Security Applications. in BVKV Kumar, S Prabhakar & A Ross (eds), Proceedings of the SPIE, Biometric Technology for Human Identification V. vol. 6944, Proceedings of SPIE, vol. 6944, SPIE, Washington, pp. 1-5, SPIE Conference on Biometric Technology for Human Identification V, Orlando, United States, 18/03/08. https://doi.org/10.1117/12.778631

    A Robust Spatio-temporal Face Modelling Approach Using 3D Multimodal Fusion for Biometric Security Applications. / Chetty, Girija; Wagner, Michael.

    Proceedings of the SPIE, Biometric Technology for Human Identification V. ed. / B.V.K Vijaya Kumar; Salil Prabhakar; Arun Ross. Vol. 6944 Washington : SPIE, 2008. p. 1-5 (Proceedings of SPIE; Vol. 6944).

    Research output: A Conference proceeding or a Chapter in BookConference contribution

    TY - GEN

    T1 - A Robust Spatio-temporal Face Modelling Approach Using 3D Multimodal Fusion for Biometric Security Applications

    AU - Chetty, Girija

    AU - Wagner, Michael

    PY - 2008

    Y1 - 2008

    N2 - In this paper, we propose a robust spatio-temporal face modelling approach based on multilevel fusion strategy involving cascaded fusion of hybrid multimodal fusion of audio-lip-face motion, correlation and depth features for biometric security application. The proposed approach combines the information from different audio-video based modules, namely: audio-lip motion module, audio-lip correlation module, 2D+3D motion-depth fusion module, and performs a hybrid cascaded fusion in an automatic, unsupervised and adaptive manner, by adapting to the local performance of each module. This is done by taking the output-score based reliability estimates (confidence measures) of each of the module into account. The module weightings are determined automatically such that the reliability measure of the combined scores is maximised. To test the robustness of the proposed approach, the audio and visual speech (mouth) modalities are degraded to emulate various levels of train/test mismatch; employing additive white Gaussian noise for the audio and JPEG compression for the video signals. The results show improved fusion performance for a range of tested levels of audio and video degradation, compared to the individual module performances. Experiments on a 3D stereovision database AVOZES show that, at severe levels of audio and video mismatch, the audio, mouth, 3D face, and tri-module (audio-lip motion, correlation and depth) fusion EERs were 42.9%, 32%, 15%, and 7.3% respectively for biometric identity verification scenario

    AB - In this paper, we propose a robust spatio-temporal face modelling approach based on multilevel fusion strategy involving cascaded fusion of hybrid multimodal fusion of audio-lip-face motion, correlation and depth features for biometric security application. The proposed approach combines the information from different audio-video based modules, namely: audio-lip motion module, audio-lip correlation module, 2D+3D motion-depth fusion module, and performs a hybrid cascaded fusion in an automatic, unsupervised and adaptive manner, by adapting to the local performance of each module. This is done by taking the output-score based reliability estimates (confidence measures) of each of the module into account. The module weightings are determined automatically such that the reliability measure of the combined scores is maximised. To test the robustness of the proposed approach, the audio and visual speech (mouth) modalities are degraded to emulate various levels of train/test mismatch; employing additive white Gaussian noise for the audio and JPEG compression for the video signals. The results show improved fusion performance for a range of tested levels of audio and video degradation, compared to the individual module performances. Experiments on a 3D stereovision database AVOZES show that, at severe levels of audio and video mismatch, the audio, mouth, 3D face, and tri-module (audio-lip motion, correlation and depth) fusion EERs were 42.9%, 32%, 15%, and 7.3% respectively for biometric identity verification scenario

    U2 - 10.1117/12.778631

    DO - 10.1117/12.778631

    M3 - Conference contribution

    VL - 6944

    T3 - Proceedings of SPIE

    SP - 1

    EP - 5

    BT - Proceedings of the SPIE, Biometric Technology for Human Identification V

    A2 - Kumar, B.V.K Vijaya

    A2 - Prabhakar, Salil

    A2 - Ross, Arun

    PB - SPIE

    CY - Washington

    ER -

    Chetty G, Wagner M. A Robust Spatio-temporal Face Modelling Approach Using 3D Multimodal Fusion for Biometric Security Applications. In Kumar BVKV, Prabhakar S, Ross A, editors, Proceedings of the SPIE, Biometric Technology for Human Identification V. Vol. 6944. Washington: SPIE. 2008. p. 1-5. (Proceedings of SPIE). https://doi.org/10.1117/12.778631