TY - GEN
T1 - Towards Robust Mispronunciation Detection and Diagnosis for L2 English Learners with Accent-Modulating Methods
AU - Jiang, Shao Wei Fan
AU - Yan, Bi Cheng
AU - Lo, Tien Hong
AU - Chao, Fu An
AU - Chen, Berlin
N1 - Funding Information:
This research is supported in part by Chunghwa Telecom Laboratories under Grant Number TL-110-D301, and by Ministry of Science and Technology, Taiwan under Grant Number MOST 110-2634-F-008-004-through Pervasive Artificial Intelligence Research (PAIR) Labs, Taiwan, and Grant Numbers MOST 108-2221-E-003-005-MY3 and MOST 109-2221-E-003-020-MY3. Any findings and implications in the paper do not necessarily reflect those of the sponsors.
Publisher Copyright:
© 2021 IEEE.
PY - 2021
Y1 - 2021
N2 - With the acceleration of globalization, more and more people are willing or required to learn second languages (L2). One of the major remaining challenges facing current mispronunciation and diagnosis (MDD) models for use in computer-assisted pronunciation training (CAPT) is to handle speech from L2 learners with a diverse set of accents. In this paper, we set out to mitigate the adverse effects of accent variety in building an L2 English MDD system with end-to-end (E2E) neural models. To this end, we first propose an effective modeling framework that infuses accent features into an E2E MDD model, thereby making the model more accent-aware. Going a step further, we design and present disparate accent-aware modules to perform accent-aware modulation of acoustic features in a finer-grained manner, so as to enhance the discriminating capability of the resulting MDD model. Extensive sets of experiments conducted on the L2-ARCTIC benchmark dataset show the merits of our MDD model, in comparison to some existing E2E-based strong baselines and the celebrated pronunciation scoring based method.
AB - With the acceleration of globalization, more and more people are willing or required to learn second languages (L2). One of the major remaining challenges facing current mispronunciation and diagnosis (MDD) models for use in computer-assisted pronunciation training (CAPT) is to handle speech from L2 learners with a diverse set of accents. In this paper, we set out to mitigate the adverse effects of accent variety in building an L2 English MDD system with end-to-end (E2E) neural models. To this end, we first propose an effective modeling framework that infuses accent features into an E2E MDD model, thereby making the model more accent-aware. Going a step further, we design and present disparate accent-aware modules to perform accent-aware modulation of acoustic features in a finer-grained manner, so as to enhance the discriminating capability of the resulting MDD model. Extensive sets of experiments conducted on the L2-ARCTIC benchmark dataset show the merits of our MDD model, in comparison to some existing E2E-based strong baselines and the celebrated pronunciation scoring based method.
KW - accent modeling
KW - accented speech
KW - computer-assisted pronunciation training
KW - mispronunciation detection and diagnosis
KW - multi-task learning
UR - http://www.scopus.com/inward/record.url?scp=85118005928&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85118005928&partnerID=8YFLogxK
U2 - 10.1109/ASRU51503.2021.9688291
DO - 10.1109/ASRU51503.2021.9688291
M3 - Conference contribution
AN - SCOPUS:85118005928
T3 - 2021 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2021 - Proceedings
SP - 1065
EP - 1070
BT - 2021 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2021 - Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2021 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2021
Y2 - 13 December 2021 through 17 December 2021
ER -