SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
Liang-Yuan Wu, Zih-Ching Chen, Tongshuang Wu, Chao-Han Huck Yang, Hua Shen
Read on arXiv →
Key claim
Current models struggle with understanding emotional cues in dialogue.
In plain English
Imagine you're building a conversational AI that can chat with people just like a human would. A big part of that is understanding not just the words being said, but also the emotions and social cues behind them. This is where things get tricky. Current systems often evaluate emotional intelligence by looking at text alone or just the sound of speech, missing the bigger picture of how these elements work together in real conversations. This leads to problems like misunderstanding the emotional tone of a dialogue or failing to respond appropriately in a multi-turn exchange. These issues are known as 'modality shortcuts' and 'contextual amnesia.'
To tackle these challenges, the authors introduce a new framework called SpeechEQ. This framework is designed to evaluate how well speech-language models understand and respond to emotional cues in conversations. It includes a dataset of over 2,200 dialogues that cover various emotional aspects, allowing for a more comprehensive assessment of AI's emotional intelligence. The findings reveal that while some models perform better than others, they still struggle with the nuances of human emotion, often defaulting to text-based interpretations. This means that if you're building a conversational AI, you need to be aware that current models might not fully grasp the emotional context, which can lead to misunderstandings in dialogue.
In practical terms, this framework provides a new benchmark for evaluating emotional intelligence in AI, helping developers identify where their systems might fall short and guiding improvements in future models.
The framework introduces a new way to evaluate emotional intelligence in multimodal conversational systems.
The experiments are based on a validated dataset and provide insights into existing model limitations.
Deep reliability assessment
The methodology supports a controlled diagnostic claim: current speech-language models often fail to use paralinguistic cues consistently in synthetic, forced-choice, multi-turn spoken dialogues. It overreaches if interpreted as measuring real-world emotional intelligence, because the dataset is LLM/TTS-generated, culturally narrow unless otherwise validated, and SEQ is cohort-normalized rather than an absolute human-like EQ measure.
Reproducibility
Dataset appears available: https://huggingface.co/datasets/SpeechEQ/SpeechEQ. A demo page is also provided at https://binomial14.github.io/speecheq-demo/, but no code repository is mentioned in the provided text.
Key figure
Figure 1 shows the SPEECHEQ dataset construction pipeline: EQ-i 2.0 scenario generation, dialogue generation, tone candidate generation, tone filtering for resonant versus dissonant deliveries, and final TTS synthesis.