TY - GEN
T1 - Online Human Action Recognition Using Deep Learning for Indoor Smart Mobile Robots
AU - Hsieh, Jih Tang
AU - Chiang, Meng Lin
AU - Fang, Chiung Yao
AU - Chen, Sei-Wang
N1 - Funding Information:
ACKNOWLEDGMENT The authors would like to thank the Ministry of Science and Technology of the Republic of China, Taiwan for financially supporting this research under Contract No. MOST-108-2221-E-003-015 and MOST 109-2221-E-003-021. We would like to thank Uni-edit (www.uni-edit.net) for editing and proofreading this manuscript.
Publisher Copyright:
© 2021 IEEE.
PY - 2021/2/19
Y1 - 2021/2/19
N2 - This research proposes a vision-based online human action recognition system. This system uses deep learning methods to recognise human action under moving camera circumstances. The proposed system consists of five stages: human detection, human tracking, feature extraction, action classification and fusion. The system uses three kinds of input information: colour intensity, short-term dynamic information and skeletal joints. In the human detection stage, a two-dimensional (2D) pose estimator method is used to detect a human. In the human tracking stage, a deep SORT tracking method is used to track the human. In the feature extraction stage, three kinds of features, spatial, temporal and structural, are extracted to analyse human actions. In the action classification stage, three kinds of features of human actions are respectively classified by three kinds of long short-term memory (LSTM) classifiers. In the fusion stage, a fusion method is used to leverage the three output results from the LSTM classifiers. This study constructs a computer vision and image understanding (CVIU) Moving Camera Human Action dataset (CVIU dataset), containing 3, 646 human action sequences, including 11 types of single human actions and 5 types of interactive human actions. This dataset was used to train and evaluate the proposed system. Experimental results showed that the recognition rates of spatial features, temporal features and structural features were 96.64%, 81.87% and 68.10%, respectively. Finally, the fusion result of human action recognition for indoor smart mobile robots in this study was 96.84%.
AB - This research proposes a vision-based online human action recognition system. This system uses deep learning methods to recognise human action under moving camera circumstances. The proposed system consists of five stages: human detection, human tracking, feature extraction, action classification and fusion. The system uses three kinds of input information: colour intensity, short-term dynamic information and skeletal joints. In the human detection stage, a two-dimensional (2D) pose estimator method is used to detect a human. In the human tracking stage, a deep SORT tracking method is used to track the human. In the feature extraction stage, three kinds of features, spatial, temporal and structural, are extracted to analyse human actions. In the action classification stage, three kinds of features of human actions are respectively classified by three kinds of long short-term memory (LSTM) classifiers. In the fusion stage, a fusion method is used to leverage the three output results from the LSTM classifiers. This study constructs a computer vision and image understanding (CVIU) Moving Camera Human Action dataset (CVIU dataset), containing 3, 646 human action sequences, including 11 types of single human actions and 5 types of interactive human actions. This dataset was used to train and evaluate the proposed system. Experimental results showed that the recognition rates of spatial features, temporal features and structural features were 96.64%, 81.87% and 68.10%, respectively. Finally, the fusion result of human action recognition for indoor smart mobile robots in this study was 96.84%.
KW - Deep learning
KW - Indoor smart mobile robot
KW - Long short-term memory
KW - Online human action recognition
UR - http://www.scopus.com/inward/record.url?scp=85104646258&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85104646258&partnerID=8YFLogxK
U2 - 10.1109/ICCCIS51004.2021.9397242
DO - 10.1109/ICCCIS51004.2021.9397242
M3 - Conference contribution
AN - SCOPUS:85104646258
T3 - Proceedings - IEEE 2021 International Conference on Computing, Communication, and Intelligent Systems, ICCCIS 2021
SP - 425
EP - 433
BT - Proceedings - IEEE 2021 International Conference on Computing, Communication, and Intelligent Systems, ICCCIS 2021
A2 - Astya, Parma Nand
A2 - Singh, Manjeet
A2 - Roy, Nihar Ranjan
A2 - Raj, Gaurav
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2021 IEEE International Conference on Computing, Communication, and Intelligent Systems, ICCCIS 2021
Y2 - 19 February 2021 through 20 February 2021
ER -
