Modulation spectrum augmentation for robust speech recognition

Bi Cheng Yan*, Shih Hung Liu, Berlin Chen

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Data augmentation is a crucial mechanism being employed to increase the diversity of training data in order to avoid overfitting and improve robustness of statistical models in various applications. In the context of automatic speech recognition (ASR), a recent trend has been to develop effective methods to augment training speech data by warping or masking utterances based on their waveforms or spectrograms. Extending this line of research, we make attempts to explore novel ways to generate augmented training speech data, in comparison to the existing state-of-the-art approaches. The main contribution of this paper is at least two-fold. First, we propose to warp the intermediate representation of the cepstral feature vector sequence of an utterance in a holistic manner. This intermediate representation can be embodied in different modulation domains by performing discrete Fourier transform (DFT) along the either the time- or the component-axis of a cepstral feature vector sequence. Second, we also develop a two-stage augmentation approach, which successively conduct perturbation in the waveform domain and warping in different modulation domains of cepstral speech feature vector sequences, to further enhance robustness. A series of experiments are carried out on the Aurora-4 database and task, in conjunction with a typical DNN-HMM based ASR system. The proposed augmentation method that conducts warping in the component-axis modulation domain of cepstral feature vector sequences can yield a word error rate reduction (WERR) of 17.6% and 0.69%, respectively, for the clean- and multi-condition training settings. In addition, the proposed two-stage augmentation method can at best achieve a WERR of 1.13% when using the multi-condition training setup.

Original languageEnglish
Title of host publicationProceedings of the International Conference on Advanced Information Science and System, AISS 2019
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450372916
Publication statusPublished - 2019 Nov 15
Event2019 International Conference on Advanced Information Science and System, AISS 2019 - Singapore, Singapore
Duration: 2019 Nov 152019 Nov 17

Publication series

NameACM International Conference Proceeding Series


Conference2019 International Conference on Advanced Information Science and System, AISS 2019


  • Data augmentation
  • Modulation spectra
  • Robustness
  • Speech recognition

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications


Dive into the research topics of 'Modulation spectrum augmentation for robust speech recognition'. Together they form a unique fingerprint.

Cite this