Real-Time Voice AI Hears but Does Not Listen
Martijn Bartelds, Federico Bianchi, James Zou
Read on arXiv →
Key claim
Voice AI often ignores emotional cues despite recognizing them.
In plain English
Imagine you're building a customer support system that uses voice AI to handle calls. You want it to understand not just what people say, but how they say it — their tone, emotion, and urgency. Currently, many voice AI systems focus primarily on the words spoken, often missing important emotional cues. For instance, they might end a call with someone who is crying, thinking everything is fine, or approve a transaction when the caller sounds scared. This disconnect happens even though the systems can recognize these emotions when asked directly. This is what's called the emotional intelligence gap of voice AI. It means that while the systems can identify distress or sarcasm, they often ignore these signals when making decisions, leading to potentially harmful outcomes. The paper suggests that simply prompting these systems to pay attention to vocal delivery doesn't fully solve the problem, as improvements are inconsistent. This highlights a significant limitation in current voice AI technology, indicating that developers should be cautious about relying on these systems in sensitive situations where tone and emotion matter.
The paper identifies a significant gap in how voice AI systems interpret emotional cues, which is a meaningful extension of existing work.
The findings are based on evaluations of multiple leading systems across various scenarios, providing solid evidence for the claims made.
Deep reliability assessment
The methodology supports the claim that, in deliberately conflictual speech examples, the tested production realtime voice systems often privilege lexical content over vocal delivery, and that this can persist even when they can identify the vocal cue in a separate probe. It would be overclaiming to conclude that these systems generally lack emotional intelligence across all real-world voice interactions, since the evidence is scenario-driven and adversarial rather than a broad deployment study.
Reproducibility
No open-source code or dataset release is evident from the provided text. The paper lists a project page at https://real-time-voice.github.io, but the excerpt does not confirm whether prompts, audio stimuli, annotations, or evaluation scripts are released.
Key figure
Figure 1 summarizes five conflict tests where the spoken words point to one answer or action while the vocal delivery or speaker property points to another, and shows that GPT Realtime 2, Gemini 3.1 Flash Live, and Qwen3.5 Omni variants usually follow the words rather than the voice.