Lightweight Spatio-Temporal Convolutional Neural Network for Audio-Visual Emotion Recognition

Emotion recognition (ER) enhances human-computer interaction in customer service and healthcare. However, the high computational complexity of existing convolutional neural network (CNN)-based approaches limits their real-time applicability in compact audio-visual emotion recognition (AVER) systems. This study introduces a lightweight deep learning-based AVER framework, employing a 2-layer, 1D CNN for audio analysis and a 3-layer, 2D CNN for facial image processing. The spatial model (2D CNN) reduces input complexity by using grayscale images, downsizing to 64 × 64 pixels, and performing three convolutions to extract facial patches. For audio, 1D convolution on Mel-Frequency Cepstral Coefficients (MFCCs) helps preserve essential features while lowering computational demand. With small kernel size and set of optimized parameters, the proposed framework balances performance and complexity. Benchmarking on SAVEE, RAVDESS, and MEAD datasets show accuracy of 97.57%, 95.89%, and 98.57%, respectively, demonstrating its potential for resource-constrained devices. Integrated gradient analysis further reveals key dependencies on eyebrows, eyes, and mouth for facial ER, and 40 MFCCs for audio ER.