Extending long short-term memory for multi-view structured learning

Shyam Rajagopalan, Louis-Philippe Morency, Tadas Baltrusaitis, Roland GOECKE

Research output: A Conference proceeding or a Chapter in BookConference contributionpeer-review

87 Citations (Scopus)
1 Downloads (Pure)


Long Short-Term Memory (LSTM) networks have been successfully applied to a number of sequence learning problems but they lack the design flexibility to model multiple view interactions, limiting their ability to exploit multi-view relationships. In this paper, we propose a Multi-View LSTM (MV-LSTM), which explicitly models the view-specific and cross-view interactions over time or structured outputs. We evaluate the MV-LSTM model on four publicly available datasets spanning two very different structured learning problems: multimodal behaviour recognition and image captioning. The experimental results show competitive performance on all four datasets when compared with state-of-the-art models.
Original languageEnglish
Title of host publicationProceedings of the European Conference on Computer Vision (ECCV 2016)
EditorsBastian Leibe, Jiri Matas, Nicu Sebe, Max Welling
Place of PublicationCham
Number of pages16
ISBN (Print)9783319464770
Publication statusPublished - 2016
EventThe European Conference on Computer Vision - Amsterdam, Amsterdam, Netherlands
Duration: 8 Oct 201616 Oct 2016

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9911 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceThe European Conference on Computer Vision
Abbreviated titleECCV 2016


Dive into the research topics of 'Extending long short-term memory for multi-view structured learning'. Together they form a unique fingerprint.

Cite this