Extending long short-term memory for multi-view structured learning

Shyam Rajagopalan; Louis-Philippe Morency; Tadas Baltrusaitis; Roland GOECKE

doi:10.1007/978-3-319-46478-7_21

Extending long short-term memory for multi-view structured learning

Shyam Rajagopalan, Louis-Philippe Morency, Tadas Baltrusaitis, Roland GOECKE

Information Systems

Research output: A Conference proceeding or a Chapter in Book › Conference contribution › peer-review

90 Citations (Scopus)

1 Downloads (Pure)

Abstract

Long Short-Term Memory (LSTM) networks have been successfully applied to a number of sequence learning problems but they lack the design flexibility to model multiple view interactions, limiting their ability to exploit multi-view relationships. In this paper, we propose a Multi-View LSTM (MV-LSTM), which explicitly models the view-specific and cross-view interactions over time or structured outputs. We evaluate the MV-LSTM model on four publicly available datasets spanning two very different structured learning problems: multimodal behaviour recognition and image captioning. The experimental results show competitive performance on all four datasets when compared with state-of-the-art models.

Original language	English
Title of host publication	Proceedings of the European Conference on Computer Vision (ECCV 2016)
Editors	Bastian Leibe, Jiri Matas, Nicu Sebe, Max Welling
Place of Publication	Cham
Publisher	Springer
Pages	338-353
Number of pages	16
Volume	9911
ISBN (Print)	9783319464770
DOIs	https://doi.org/10.1007/978-3-319-46478-7_21
Publication status	Published - 2016
Event	The European Conference on Computer Vision - Amsterdam, Amsterdam, Netherlands Duration: 8 Oct 2016 → 16 Oct 2016

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	9911 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	The European Conference on Computer Vision
Abbreviated title	ECCV 2016
Country/Territory	Netherlands
City	Amsterdam
Period	8/10/16 → 16/10/16

Access to Document

10.1007/978-3-319-46478-7_21

Cite this

Rajagopalan, S., Morency, L-P., Baltrusaitis, T., & GOECKE, R. (2016). Extending long short-term memory for multi-view structured learning. In B. Leibe, J. Matas, N. Sebe, & M. Welling (Eds.), Proceedings of the European Conference on Computer Vision (ECCV 2016) (Vol. 9911, pp. 338-353). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9911 LNCS). Springer. https://doi.org/10.1007/978-3-319-46478-7_21

Rajagopalan, Shyam ; Morency, Louis-Philippe ; Baltrusaitis, Tadas et al. / Extending long short-term memory for multi-view structured learning. Proceedings of the European Conference on Computer Vision (ECCV 2016). editor / Bastian Leibe ; Jiri Matas ; Nicu Sebe ; Max Welling. Vol. 9911 Cham : Springer, 2016. pp. 338-353 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{530c8432b9184d25bdfa4e90af747706,

title = "Extending long short-term memory for multi-view structured learning",

abstract = "Long Short-Term Memory (LSTM) networks have been successfully applied to a number of sequence learning problems but they lack the design flexibility to model multiple view interactions, limiting their ability to exploit multi-view relationships. In this paper, we propose a Multi-View LSTM (MV-LSTM), which explicitly models the view-specific and cross-view interactions over time or structured outputs. We evaluate the MV-LSTM model on four publicly available datasets spanning two very different structured learning problems: multimodal behaviour recognition and image captioning. The experimental results show competitive performance on all four datasets when compared with state-of-the-art models.",

keywords = "Long Short-Term Memory, Multi-View Learning, Behaviour recognition, Image Caption, Image caption, Long short-term memory, Multi-view learning",

author = "Shyam Rajagopalan and Louis-Philippe Morency and Tadas Baltrusaitis and Roland GOECKE",

year = "2016",

doi = "10.1007/978-3-319-46478-7_21",

language = "English",

isbn = "9783319464770",

volume = "9911",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "338--353",

editor = "Bastian Leibe and Jiri Matas and Nicu Sebe and Max Welling",

booktitle = "Proceedings of the European Conference on Computer Vision (ECCV 2016)",

address = "Netherlands",

note = "The European Conference on Computer Vision, ECCV 2016 ; Conference date: 08-10-2016 Through 16-10-2016",

}

Rajagopalan, S, Morency, L-P, Baltrusaitis, T & GOECKE, R 2016, Extending long short-term memory for multi-view structured learning. in B Leibe, J Matas, N Sebe & M Welling (eds), Proceedings of the European Conference on Computer Vision (ECCV 2016). vol. 9911, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9911 LNCS, Springer, Cham, pp. 338-353, The European Conference on Computer Vision, Amsterdam, Netherlands, 8/10/16. https://doi.org/10.1007/978-3-319-46478-7_21

Extending long short-term memory for multi-view structured learning. / Rajagopalan, Shyam; Morency, Louis-Philippe; Baltrusaitis, Tadas et al.
Proceedings of the European Conference on Computer Vision (ECCV 2016). ed. / Bastian Leibe; Jiri Matas; Nicu Sebe; Max Welling. Vol. 9911 Cham: Springer, 2016. p. 338-353 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9911 LNCS).

Research output: A Conference proceeding or a Chapter in Book › Conference contribution › peer-review

TY - GEN

T1 - Extending long short-term memory for multi-view structured learning

AU - Rajagopalan, Shyam

AU - Morency, Louis-Philippe

AU - Baltrusaitis, Tadas

AU - GOECKE, Roland

PY - 2016

Y1 - 2016

N2 - Long Short-Term Memory (LSTM) networks have been successfully applied to a number of sequence learning problems but they lack the design flexibility to model multiple view interactions, limiting their ability to exploit multi-view relationships. In this paper, we propose a Multi-View LSTM (MV-LSTM), which explicitly models the view-specific and cross-view interactions over time or structured outputs. We evaluate the MV-LSTM model on four publicly available datasets spanning two very different structured learning problems: multimodal behaviour recognition and image captioning. The experimental results show competitive performance on all four datasets when compared with state-of-the-art models.

AB - Long Short-Term Memory (LSTM) networks have been successfully applied to a number of sequence learning problems but they lack the design flexibility to model multiple view interactions, limiting their ability to exploit multi-view relationships. In this paper, we propose a Multi-View LSTM (MV-LSTM), which explicitly models the view-specific and cross-view interactions over time or structured outputs. We evaluate the MV-LSTM model on four publicly available datasets spanning two very different structured learning problems: multimodal behaviour recognition and image captioning. The experimental results show competitive performance on all four datasets when compared with state-of-the-art models.

KW - Long Short-Term Memory

KW - Multi-View Learning

KW - Behaviour recognition

KW - Image Caption

KW - Image caption

KW - Long short-term memory

KW - Multi-view learning

UR - http://www.scopus.com/inward/record.url?scp=84990020972&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-46478-7_21

DO - 10.1007/978-3-319-46478-7_21

M3 - Conference contribution

SN - 9783319464770

VL - 9911

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 338

EP - 353

BT - Proceedings of the European Conference on Computer Vision (ECCV 2016)

A2 - Leibe, Bastian

A2 - Matas, Jiri

A2 - Sebe, Nicu

A2 - Welling, Max

PB - Springer

CY - Cham

T2 - The European Conference on Computer Vision

Y2 - 8 October 2016 through 16 October 2016

ER -

Rajagopalan S, Morency L-P, Baltrusaitis T, GOECKE R. Extending long short-term memory for multi-view structured learning. In Leibe B, Matas J, Sebe N, Welling M, editors, Proceedings of the European Conference on Computer Vision (ECCV 2016). Vol. 9911. Cham: Springer. 2016. p. 338-353. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-46478-7_21

Extending long short-term memory for multi-view structured learning

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this