Extending long short-term memory for multi-view structured learning

Shyam Rajagopalan, Louis-Philippe Morency, Tadas Baltrusaitis, Roland GOECKE

Research output: A Conference proceeding or a Chapter in BookConference contribution

22 Citations (Scopus)
1 Downloads (Pure)

Abstract

Long Short-Term Memory (LSTM) networks have been successfully applied to a number of sequence learning problems but they lack the design flexibility to model multiple view interactions, limiting their ability to exploit multi-view relationships. In this paper, we propose a Multi-View LSTM (MV-LSTM), which explicitly models the view-specific and cross-view interactions over time or structured outputs. We evaluate the MV-LSTM model on four publicly available datasets spanning two very different structured learning problems: multimodal behaviour recognition and image captioning. The experimental results show competitive performance on all four datasets when compared with state-of-the-art models.
Original languageEnglish
Title of host publicationProceedings of the European Conference on Computer Vision (ECCV 2016)
EditorsMax Welling, Nicu Sebe, Jiri Matas, Bastian Leibe
Place of PublicationCham
PublisherSpringer
Pages338-353
Number of pages16
Volume9911
ISBN (Print)9783319464770
DOIs
Publication statusPublished - 2016
EventThe European Conference on Computer Vision - Amsterdam, Amsterdam, Netherlands
Duration: 8 Oct 201616 Oct 2016

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9911 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceThe European Conference on Computer Vision
Abbreviated titleECCV 2016
CountryNetherlands
CityAmsterdam
Period8/10/1616/10/16

Fingerprint Dive into the research topics of 'Extending long short-term memory for multi-view structured learning'. Together they form a unique fingerprint.

  • Cite this

    Rajagopalan, S., Morency, L-P., Baltrusaitis, T., & GOECKE, R. (2016). Extending long short-term memory for multi-view structured learning. In M. Welling, N. Sebe, J. Matas, & B. Leibe (Eds.), Proceedings of the European Conference on Computer Vision (ECCV 2016) (Vol. 9911, pp. 338-353). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9911 LNCS). Springer. https://doi.org/10.1007/978-3-319-46478-7_21