Extending long short-term memory for multi-view structured learning

Shyam Rajagopalan, Louis-Philippe Morency, Tadas Baltrusaitis, Roland GOECKE

Research output: A Conference proceeding or a Chapter in BookConference contribution

8 Citations (Scopus)
1 Downloads (Pure)

Abstract

Long Short-Term Memory (LSTM) networks have been successfully applied to a number of sequence learning problems but they lack the design flexibility to model multiple view interactions, limiting their ability to exploit multi-view relationships. In this paper, we propose a Multi-View LSTM (MV-LSTM), which explicitly models the view-specific and cross-view interactions over time or structured outputs. We evaluate the MV-LSTM model on four publicly available datasets spanning two very different structured learning problems: multimodal behaviour recognition and image captioning. The experimental results show competitive performance on all four datasets when compared with state-of-the-art models.
Original languageEnglish
Title of host publicationProceedings of the European Conference on Computer Vision (ECCV 2016)
EditorsMax Welling, Nicu Sebe, Jiri Matas, Bastian Leibe
Place of PublicationCham
PublisherSpringer
Pages338-353
Number of pages16
Volume9911
ISBN (Print)9783319464770
DOIs
Publication statusPublished - 2016
EventThe European Conference on Computer Vision - Amsterdam, Amsterdam, Netherlands
Duration: 8 Oct 201616 Oct 2016

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9911 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceThe European Conference on Computer Vision
Abbreviated titleECCV 2016
CountryNetherlands
CityAmsterdam
Period8/10/1616/10/16

Fingerprint

Long short-term memory

Cite this

Rajagopalan, S., Morency, L-P., Baltrusaitis, T., & GOECKE, R. (2016). Extending long short-term memory for multi-view structured learning. In M. Welling, N. Sebe, J. Matas, & B. Leibe (Eds.), Proceedings of the European Conference on Computer Vision (ECCV 2016) (Vol. 9911, pp. 338-353). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9911 LNCS). Cham: Springer. https://doi.org/10.1007/978-3-319-46478-7_21
Rajagopalan, Shyam ; Morency, Louis-Philippe ; Baltrusaitis, Tadas ; GOECKE, Roland. / Extending long short-term memory for multi-view structured learning. Proceedings of the European Conference on Computer Vision (ECCV 2016). editor / Max Welling ; Nicu Sebe ; Jiri Matas ; Bastian Leibe. Vol. 9911 Cham : Springer, 2016. pp. 338-353 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{530c8432b9184d25bdfa4e90af747706,
title = "Extending long short-term memory for multi-view structured learning",
abstract = "Long Short-Term Memory (LSTM) networks have been successfully applied to a number of sequence learning problems but they lack the design flexibility to model multiple view interactions, limiting their ability to exploit multi-view relationships. In this paper, we propose a Multi-View LSTM (MV-LSTM), which explicitly models the view-specific and cross-view interactions over time or structured outputs. We evaluate the MV-LSTM model on four publicly available datasets spanning two very different structured learning problems: multimodal behaviour recognition and image captioning. The experimental results show competitive performance on all four datasets when compared with state-of-the-art models.",
keywords = "Long Short-Term Memory, Multi-View Learning, Behaviour recognition, Image Caption, Image caption, Long short-term memory, Multi-view learning",
author = "Shyam Rajagopalan and Louis-Philippe Morency and Tadas Baltrusaitis and Roland GOECKE",
year = "2016",
doi = "10.1007/978-3-319-46478-7_21",
language = "English",
isbn = "9783319464770",
volume = "9911",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer",
pages = "338--353",
editor = "Max Welling and Nicu Sebe and Jiri Matas and Bastian Leibe",
booktitle = "Proceedings of the European Conference on Computer Vision (ECCV 2016)",
address = "Netherlands",

}

Rajagopalan, S, Morency, L-P, Baltrusaitis, T & GOECKE, R 2016, Extending long short-term memory for multi-view structured learning. in M Welling, N Sebe, J Matas & B Leibe (eds), Proceedings of the European Conference on Computer Vision (ECCV 2016). vol. 9911, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9911 LNCS, Springer, Cham, pp. 338-353, The European Conference on Computer Vision, Amsterdam, Netherlands, 8/10/16. https://doi.org/10.1007/978-3-319-46478-7_21

Extending long short-term memory for multi-view structured learning. / Rajagopalan, Shyam; Morency, Louis-Philippe; Baltrusaitis, Tadas; GOECKE, Roland.

Proceedings of the European Conference on Computer Vision (ECCV 2016). ed. / Max Welling; Nicu Sebe; Jiri Matas; Bastian Leibe. Vol. 9911 Cham : Springer, 2016. p. 338-353 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9911 LNCS).

Research output: A Conference proceeding or a Chapter in BookConference contribution

TY - GEN

T1 - Extending long short-term memory for multi-view structured learning

AU - Rajagopalan, Shyam

AU - Morency, Louis-Philippe

AU - Baltrusaitis, Tadas

AU - GOECKE, Roland

PY - 2016

Y1 - 2016

N2 - Long Short-Term Memory (LSTM) networks have been successfully applied to a number of sequence learning problems but they lack the design flexibility to model multiple view interactions, limiting their ability to exploit multi-view relationships. In this paper, we propose a Multi-View LSTM (MV-LSTM), which explicitly models the view-specific and cross-view interactions over time or structured outputs. We evaluate the MV-LSTM model on four publicly available datasets spanning two very different structured learning problems: multimodal behaviour recognition and image captioning. The experimental results show competitive performance on all four datasets when compared with state-of-the-art models.

AB - Long Short-Term Memory (LSTM) networks have been successfully applied to a number of sequence learning problems but they lack the design flexibility to model multiple view interactions, limiting their ability to exploit multi-view relationships. In this paper, we propose a Multi-View LSTM (MV-LSTM), which explicitly models the view-specific and cross-view interactions over time or structured outputs. We evaluate the MV-LSTM model on four publicly available datasets spanning two very different structured learning problems: multimodal behaviour recognition and image captioning. The experimental results show competitive performance on all four datasets when compared with state-of-the-art models.

KW - Long Short-Term Memory

KW - Multi-View Learning

KW - Behaviour recognition

KW - Image Caption

KW - Image caption

KW - Long short-term memory

KW - Multi-view learning

UR - http://www.scopus.com/inward/record.url?scp=84990020972&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-46478-7_21

DO - 10.1007/978-3-319-46478-7_21

M3 - Conference contribution

SN - 9783319464770

VL - 9911

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 338

EP - 353

BT - Proceedings of the European Conference on Computer Vision (ECCV 2016)

A2 - Welling, Max

A2 - Sebe, Nicu

A2 - Matas, Jiri

A2 - Leibe, Bastian

PB - Springer

CY - Cham

ER -

Rajagopalan S, Morency L-P, Baltrusaitis T, GOECKE R. Extending long short-term memory for multi-view structured learning. In Welling M, Sebe N, Matas J, Leibe B, editors, Proceedings of the European Conference on Computer Vision (ECCV 2016). Vol. 9911. Cham: Springer. 2016. p. 338-353. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-46478-7_21