Learning depth from monocular sequence with convolutional LSTM network

Chia Hung Yeh; Yao Pao Huang; Chih Yang Lin; Min Hui Lin

doi:10.1007/978-3-030-29029-0_48

Learning depth from monocular sequence with convolutional LSTM network

Chia Hung Yeh^*, Yao Pao Huang, Chih Yang Lin, Min Hui Lin

^*Corresponding author for this work

Department of Electrical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

Abstract

Resolving depth from monocular RGB image has been a long-standing task in computer vision and robotics. Recently, deep learning based methods has become a popular algorithm on depth estimation. Most existing learning based methods take image-pair as input and utilize feature matching across frames to resolve depth. However, two-frame methods require sufficient and static camera motion to reach optimal performance, while camera motion is usually uncontrollable in most application scenarios. In this paper we propose a recurrent neural network based depth estimation network. With the ability of taking multiple images as input, recurrent neural network will decide by itself which image to reference during estimation. We train a u-net like network architecture which utilizes convolutional LSTM in the encoder. We demonstrate our proposed method with the TUM RGB-D dataset, where our proposed method shows the ability of estimating depth with various sequence lengths as input.

Original language	English
Title of host publication	Advances in Networked-based Information Systems - The 22nd International Conference on Network-Based Information Systems, NBiS 2019
Editors	Leonard Barolli, Hiroaki Nishino, Tomoya Enokido, Makoto Takizawa
Publisher	Springer Verlag
Pages	502-507
Number of pages	6
ISBN (Print)	9783030290283
DOIs	https://doi.org/10.1007/978-3-030-29029-0_48
Publication status	Published - 2020
Event	22nd International Conference on Network-Based Information Systems, NBiS 2019 - Oita, Japan Duration: 2019 Sept 5 → 2019 Sept 7

Publication series

Name	Advances in Intelligent Systems and Computing
Volume	1036
ISSN (Print)	2194-5357
ISSN (Electronic)	2194-5365

Conference

Conference	22nd International Conference on Network-Based Information Systems, NBiS 2019
Country/Territory	Japan
City	Oita
Period	2019/09/05 → 2019/09/07

Keywords

Convolutional LSTM
Deep learning
Multi-view depth estimation
Recurrent neural network

ASJC Scopus subject areas

Control and Systems Engineering
General Computer Science

Access to Document

10.1007/978-3-030-29029-0_48

Cite this

Yeh, C. H., Huang, Y. P., Lin, C. Y., & Lin, M. H. (2020). Learning depth from monocular sequence with convolutional LSTM network. In L. Barolli, H. Nishino, T. Enokido, & M. Takizawa (Eds.), Advances in Networked-based Information Systems - The 22nd International Conference on Network-Based Information Systems, NBiS 2019 (pp. 502-507). (Advances in Intelligent Systems and Computing; Vol. 1036). Springer Verlag. https://doi.org/10.1007/978-3-030-29029-0_48

Learning depth from monocular sequence with convolutional LSTM network. / Yeh, Chia Hung; Huang, Yao Pao; Lin, Chih Yang et al.
Advances in Networked-based Information Systems - The 22nd International Conference on Network-Based Information Systems, NBiS 2019. ed. / Leonard Barolli; Hiroaki Nishino; Tomoya Enokido; Makoto Takizawa. Springer Verlag, 2020. p. 502-507 (Advances in Intelligent Systems and Computing; Vol. 1036).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Yeh, CH, Huang, YP, Lin, CY & Lin, MH 2020, Learning depth from monocular sequence with convolutional LSTM network. in L Barolli, H Nishino, T Enokido & M Takizawa (eds), Advances in Networked-based Information Systems - The 22nd International Conference on Network-Based Information Systems, NBiS 2019. Advances in Intelligent Systems and Computing, vol. 1036, Springer Verlag, pp. 502-507, 22nd International Conference on Network-Based Information Systems, NBiS 2019, Oita, Japan, 2019/09/05. https://doi.org/10.1007/978-3-030-29029-0_48

Yeh CH, Huang YP, Lin CY, Lin MH. Learning depth from monocular sequence with convolutional LSTM network. In Barolli L, Nishino H, Enokido T, Takizawa M, editors, Advances in Networked-based Information Systems - The 22nd International Conference on Network-Based Information Systems, NBiS 2019. Springer Verlag. 2020. p. 502-507. (Advances in Intelligent Systems and Computing). doi: 10.1007/978-3-030-29029-0_48

Yeh, Chia Hung ; Huang, Yao Pao ; Lin, Chih Yang et al. / Learning depth from monocular sequence with convolutional LSTM network. Advances in Networked-based Information Systems - The 22nd International Conference on Network-Based Information Systems, NBiS 2019. editor / Leonard Barolli ; Hiroaki Nishino ; Tomoya Enokido ; Makoto Takizawa. Springer Verlag, 2020. pp. 502-507 (Advances in Intelligent Systems and Computing).

@inproceedings{83be2a9bd7414722a025468444a1a036,

title = "Learning depth from monocular sequence with convolutional LSTM network",

abstract = "Resolving depth from monocular RGB image has been a long-standing task in computer vision and robotics. Recently, deep learning based methods has become a popular algorithm on depth estimation. Most existing learning based methods take image-pair as input and utilize feature matching across frames to resolve depth. However, two-frame methods require sufficient and static camera motion to reach optimal performance, while camera motion is usually uncontrollable in most application scenarios. In this paper we propose a recurrent neural network based depth estimation network. With the ability of taking multiple images as input, recurrent neural network will decide by itself which image to reference during estimation. We train a u-net like network architecture which utilizes convolutional LSTM in the encoder. We demonstrate our proposed method with the TUM RGB-D dataset, where our proposed method shows the ability of estimating depth with various sequence lengths as input.",

keywords = "Convolutional LSTM, Deep learning, Multi-view depth estimation, Recurrent neural network",

author = "Yeh, {Chia Hung} and Huang, {Yao Pao} and Lin, {Chih Yang} and Lin, {Min Hui}",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2020.; 22nd International Conference on Network-Based Information Systems, NBiS 2019 ; Conference date: 05-09-2019 Through 07-09-2019",

year = "2020",

doi = "10.1007/978-3-030-29029-0_48",

language = "English",

isbn = "9783030290283",

series = "Advances in Intelligent Systems and Computing",

publisher = "Springer Verlag",

pages = "502--507",

editor = "Leonard Barolli and Hiroaki Nishino and Tomoya Enokido and Makoto Takizawa",

booktitle = "Advances in Networked-based Information Systems - The 22nd International Conference on Network-Based Information Systems, NBiS 2019",

}

TY - GEN

T1 - Learning depth from monocular sequence with convolutional LSTM network

AU - Yeh, Chia Hung

AU - Huang, Yao Pao

AU - Lin, Chih Yang

AU - Lin, Min Hui

N1 - Publisher Copyright: © Springer Nature Switzerland AG 2020.

PY - 2020

Y1 - 2020

N2 - Resolving depth from monocular RGB image has been a long-standing task in computer vision and robotics. Recently, deep learning based methods has become a popular algorithm on depth estimation. Most existing learning based methods take image-pair as input and utilize feature matching across frames to resolve depth. However, two-frame methods require sufficient and static camera motion to reach optimal performance, while camera motion is usually uncontrollable in most application scenarios. In this paper we propose a recurrent neural network based depth estimation network. With the ability of taking multiple images as input, recurrent neural network will decide by itself which image to reference during estimation. We train a u-net like network architecture which utilizes convolutional LSTM in the encoder. We demonstrate our proposed method with the TUM RGB-D dataset, where our proposed method shows the ability of estimating depth with various sequence lengths as input.

AB - Resolving depth from monocular RGB image has been a long-standing task in computer vision and robotics. Recently, deep learning based methods has become a popular algorithm on depth estimation. Most existing learning based methods take image-pair as input and utilize feature matching across frames to resolve depth. However, two-frame methods require sufficient and static camera motion to reach optimal performance, while camera motion is usually uncontrollable in most application scenarios. In this paper we propose a recurrent neural network based depth estimation network. With the ability of taking multiple images as input, recurrent neural network will decide by itself which image to reference during estimation. We train a u-net like network architecture which utilizes convolutional LSTM in the encoder. We demonstrate our proposed method with the TUM RGB-D dataset, where our proposed method shows the ability of estimating depth with various sequence lengths as input.

KW - Convolutional LSTM

KW - Deep learning

KW - Multi-view depth estimation

KW - Recurrent neural network

UR - http://www.scopus.com/inward/record.url?scp=85071496821&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85071496821&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-29029-0_48

DO - 10.1007/978-3-030-29029-0_48

M3 - Conference contribution

AN - SCOPUS:85071496821

SN - 9783030290283

T3 - Advances in Intelligent Systems and Computing

SP - 502

EP - 507

BT - Advances in Networked-based Information Systems - The 22nd International Conference on Network-Based Information Systems, NBiS 2019

A2 - Barolli, Leonard

A2 - Nishino, Hiroaki

A2 - Enokido, Tomoya

A2 - Takizawa, Makoto

PB - Springer Verlag

T2 - 22nd International Conference on Network-Based Information Systems, NBiS 2019

Y2 - 5 September 2019 through 7 September 2019

ER -

Learning depth from monocular sequence with convolutional LSTM network

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this