Effective Cross-Utterance Language Modeling for Conversational Speech Recognition

Bi Cheng Yan, Hsin Wei Wang, Shih Hsuan Chiu, Hsuan Sheng Chiu, Berlin Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)


Conversational speech normally is embodied with loose syntactic structures at the utterance level but simultaneously exhibits topical coherence relations across consecutive utterances. Prior work has shown that capturing longer context information with a recurrent neural network or long short-term memory language model (LM) may suffer from the recent bias while excluding the long-range context. In order to capture the long-term semantic interactions among words and across utterances, we put forward disparate conversation history fusion methods for language modeling in automatic speech recognition (ASR) of conversational speech. Furthermore, a novel audio-fusion mechanism is introduced, which manages to fuse and utilize the acoustic embeddings of a current utterance and the semantic content of its corresponding conversation history in a cooperative way. To flesh out our ideas, we frame the ASR N-best hypothesis rescoring task as a prediction problem, leveraging BERT, an iconic pre-trained LM, as the ingredient vehicle to facilitate selection of the oracle hypothesis from a given N-best hypothesis list. Empirical experiments conducted on the AMI benchmark dataset seem to demonstrate the feasibility and efficacy of our methods in relation to some current top-of-line methods. The proposed methods not only achieve significant inference time reduction but also improve the ASR performance for conversational speech.

Original languageEnglish
Title of host publication2022 International Joint Conference on Neural Networks, IJCNN 2022 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781728186719
Publication statusPublished - 2022
Event2022 International Joint Conference on Neural Networks, IJCNN 2022 - Padua, Italy
Duration: 2022 Jul 182022 Jul 23

Publication series

NameProceedings of the International Joint Conference on Neural Networks


Conference2022 International Joint Conference on Neural Networks, IJCNN 2022


  • N-best hypothesis reranking
  • automatic speech recognition
  • conversational speech
  • language modeling

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence


Dive into the research topics of 'Effective Cross-Utterance Language Modeling for Conversational Speech Recognition'. Together they form a unique fingerprint.

Cite this