Exploring Branchformer-Based End-to-End Speaker Diarization with Speaker-Wise VAD Loss

Pei Ying Lee; Hau Yun Guo; Tien Hong Lo; Berlin Chen

doi:10.1109/O-COCOSDA64382.2024.10800120

Exploring Branchformer-Based End-to-End Speaker Diarization with Speaker-Wise VAD Loss

Pei Ying Lee
, Hau Yun Guo
, Tien Hong Lo
, Berlin Chen

Department of Computer Science and Information Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Speaker diarization involves partitioning an audio stream into segments according to the identity of the speaker. The encoder-decoder based attractors for the end-to-end neural diarization (EEND-EDA) model can handle overlapping speech and has shown promising performance compared to traditional methods. However, EEND-EDA fails to identify the number of speakers accurately. To address this limitation, we first replace the Transformer encoder in EEND-EDA with the Branchformer encoder. Additionally, we introduce speaker-wise VAD Loss (SAD Loss) to the self-attention mechanism of the Branchformer encoder, thereby improving the model's ability to distinguish different speakers. Extensive experimental results on the Mini-Librispeech and simulated dataset Sim2spk benchmark dataset suggest that our approach outperforms existing strong baselines by a substantial margin, achieving a significant improvement of more than 15% Diarization Error Rate (DER). We will release the source code on GitHub¹ for future research.

Original language	English
Title of host publication	2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings
Editors	Ming-Hsiang Su, Jui-Feng Yeh, Yuan-Fu Liao, Chi-Chun Lee, Yu Taso
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9798331506032
DOIs	https://doi.org/10.1109/O-COCOSDA64382.2024.10800120
Publication status	Published - 2024
Event	27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Hsinchu, Taiwan Duration: 2024 Oct 17 → 2024 Oct 19

Publication series

Name	2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings

Conference

Conference	27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024
Country/Territory	Taiwan
City	Hsinchu
Period	2024/10/17 → 2024/10/19

Keywords

auxiliary loss
branchformer
end-to-end neural di-arization
multi -head attention
speaker diarization

ASJC Scopus subject areas

Computer Vision and Pattern Recognition
Human-Computer Interaction
Information Systems
Information Systems and Management
Safety, Risk, Reliability and Quality
Library and Information Sciences
Language and Linguistics

Access to Document

10.1109/O-COCOSDA64382.2024.10800120

Cite this

Lee, P. Y., Guo, H. Y., Lo, T. H., & Chen, B. (2024). Exploring Branchformer-Based End-to-End Speaker Diarization with Speaker-Wise VAD Loss. In M.-H. Su, J.-F. Yeh, Y.-F. Liao, C.-C. Lee, & Y. Taso (Eds.), 2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings (2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/O-COCOSDA64382.2024.10800120

Exploring Branchformer-Based End-to-End Speaker Diarization with Speaker-Wise VAD Loss. / Lee, Pei Ying; Guo, Hau Yun; Lo, Tien Hong et al.
2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings. ed. / Ming-Hsiang Su; Jui-Feng Yeh; Yuan-Fu Liao; Chi-Chun Lee; Yu Taso. Institute of Electrical and Electronics Engineers Inc., 2024. (2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Lee, PY, Guo, HY, Lo, TH & Chen, B 2024, Exploring Branchformer-Based End-to-End Speaker Diarization with Speaker-Wise VAD Loss. in M-H Su, J-F Yeh, Y-F Liao, C-C Lee & Y Taso (eds), 2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings. 2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings, Institute of Electrical and Electronics Engineers Inc., 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024, Hsinchu, Taiwan, 2024/10/17. https://doi.org/10.1109/O-COCOSDA64382.2024.10800120

Lee PY, Guo HY, Lo TH, Chen B. Exploring Branchformer-Based End-to-End Speaker Diarization with Speaker-Wise VAD Loss. In Su MH, Yeh JF, Liao YF, Lee CC, Taso Y, editors, 2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2024. (2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings). doi: 10.1109/O-COCOSDA64382.2024.10800120

Lee, Pei Ying ; Guo, Hau Yun ; Lo, Tien Hong et al. / Exploring Branchformer-Based End-to-End Speaker Diarization with Speaker-Wise VAD Loss. 2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings. editor / Ming-Hsiang Su ; Jui-Feng Yeh ; Yuan-Fu Liao ; Chi-Chun Lee ; Yu Taso. Institute of Electrical and Electronics Engineers Inc., 2024. (2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings).

@inproceedings{2467634029c347b3aa1da98bf86a9900,

title = "Exploring Branchformer-Based End-to-End Speaker Diarization with Speaker-Wise VAD Loss",

abstract = "Speaker diarization involves partitioning an audio stream into segments according to the identity of the speaker. The encoder-decoder based attractors for the end-to-end neural diarization (EEND-EDA) model can handle overlapping speech and has shown promising performance compared to traditional methods. However, EEND-EDA fails to identify the number of speakers accurately. To address this limitation, we first replace the Transformer encoder in EEND-EDA with the Branchformer encoder. Additionally, we introduce speaker-wise VAD Loss (SAD Loss) to the self-attention mechanism of the Branchformer encoder, thereby improving the model's ability to distinguish different speakers. Extensive experimental results on the Mini-Librispeech and simulated dataset Sim2spk benchmark dataset suggest that our approach outperforms existing strong baselines by a substantial margin, achieving a significant improvement of more than 15\% Diarization Error Rate (DER). We will release the source code on GitHub1 for future research.",

keywords = "auxiliary loss, branchformer, end-to-end neural di-arization, multi -head attention, speaker diarization",

author = "Lee, \{Pei Ying\} and Guo, \{Hau Yun\} and Lo, \{Tien Hong\} and Berlin Chen",

note = "Publisher Copyright: {\textcopyright} 2024 IEEE.; 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 ; Conference date: 17-10-2024 Through 19-10-2024",

year = "2024",

doi = "10.1109/O-COCOSDA64382.2024.10800120",

language = "English",

series = "2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

editor = "Ming-Hsiang Su and Jui-Feng Yeh and Yuan-Fu Liao and Chi-Chun Lee and Yu Taso",

booktitle = "2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings",

}

TY - GEN

T1 - Exploring Branchformer-Based End-to-End Speaker Diarization with Speaker-Wise VAD Loss

AU - Lee, Pei Ying

AU - Guo, Hau Yun

AU - Lo, Tien Hong

AU - Chen, Berlin

PY - 2024

Y1 - 2024

N2 - Speaker diarization involves partitioning an audio stream into segments according to the identity of the speaker. The encoder-decoder based attractors for the end-to-end neural diarization (EEND-EDA) model can handle overlapping speech and has shown promising performance compared to traditional methods. However, EEND-EDA fails to identify the number of speakers accurately. To address this limitation, we first replace the Transformer encoder in EEND-EDA with the Branchformer encoder. Additionally, we introduce speaker-wise VAD Loss (SAD Loss) to the self-attention mechanism of the Branchformer encoder, thereby improving the model's ability to distinguish different speakers. Extensive experimental results on the Mini-Librispeech and simulated dataset Sim2spk benchmark dataset suggest that our approach outperforms existing strong baselines by a substantial margin, achieving a significant improvement of more than 15% Diarization Error Rate (DER). We will release the source code on GitHub1 for future research.

AB - Speaker diarization involves partitioning an audio stream into segments according to the identity of the speaker. The encoder-decoder based attractors for the end-to-end neural diarization (EEND-EDA) model can handle overlapping speech and has shown promising performance compared to traditional methods. However, EEND-EDA fails to identify the number of speakers accurately. To address this limitation, we first replace the Transformer encoder in EEND-EDA with the Branchformer encoder. Additionally, we introduce speaker-wise VAD Loss (SAD Loss) to the self-attention mechanism of the Branchformer encoder, thereby improving the model's ability to distinguish different speakers. Extensive experimental results on the Mini-Librispeech and simulated dataset Sim2spk benchmark dataset suggest that our approach outperforms existing strong baselines by a substantial margin, achieving a significant improvement of more than 15% Diarization Error Rate (DER). We will release the source code on GitHub1 for future research.

KW - auxiliary loss

KW - branchformer

KW - end-to-end neural di-arization

KW - multi -head attention

KW - speaker diarization

UR - https://www.scopus.com/pages/publications/85215693848

UR - https://www.scopus.com/pages/publications/85215693848#tab=citedBy

U2 - 10.1109/O-COCOSDA64382.2024.10800120

DO - 10.1109/O-COCOSDA64382.2024.10800120

M3 - Conference contribution

AN - SCOPUS:85215693848

T3 - 2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings

BT - 2024 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024 - Proceedings

A2 - Su, Ming-Hsiang

A2 - Yeh, Jui-Feng

A2 - Liao, Yuan-Fu

A2 - Lee, Chi-Chun

A2 - Taso, Yu

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 27th Conference on the Oriental COCOSDA International Committee for the Co-Ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2024

Y2 - 17 October 2024 through 19 October 2024

ER -

Exploring Branchformer-Based End-to-End Speaker Diarization with Speaker-Wise VAD Loss

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this