TTS evaluators miss more than naturalness

TTS evaluators often miss linguistically grounded speech errors beyond simple naturalness.

• Research org: Unspecified in arXiv abstract
• Core data: 860 utterances
• Breakthrough: 10-dimension linguistically grounded annotation schema for TTS

For years, automated text-to-speech evaluation has leaned heavily on one big proxy: naturalness. This paper argues that proxy is too blunt. Instead of asking whether a system sounds generally good, the authors break speech down into the kinds of errors listeners actually notice and then test whether automated evaluators can detect them.

The practical takeaway is simple: if you are building, tuning, or comparing TTS systems, a single MOS-style score may hide important failure modes. The paper shows that some evaluators mostly track acoustic quality, while others only catch certain issues when prompted in specific ways. That means a model can look strong on a headline score and still miss structured speech problems that matter to users.

What problem this paper is trying to fix

Automated TTS evaluation is supposed to stand in for human judgment, but human judgment is not one-dimensional. Listeners do not just hear “natural” versus “unnatural”; they notice a range of perceptual and linguistic issues. The paper’s core claim is that current evaluation tools are too coarse to reflect that reality.

That matters because TTS teams often rely on Mean Opinion Score predictors or Audio-LLM judges to speed up evaluation. If those tools mostly reward clean acoustics, they can miss pronunciation problems, prosodic issues, or other structured errors that affect usability even when the audio sounds polished at a glance.

The authors frame this as a meta-evaluation problem: not just whether a TTS model sounds good, but whether the evaluator is actually measuring the right things. That distinction is useful for developers because it changes what “better” means. A higher score is only useful if the score is sensitive to the failures you care about.

How the method works in plain English

Rather than treating naturalness as a single bucket, the paper deconstructs it into a linguistically grounded annotation schema with 10 distinct perceptual dimensions. In other words, the authors define a more detailed checklist of speech qualities and errors that trained linguist raters can apply consistently.

Using that schema, they build what they describe as the first dimension-level meta-evaluation benchmark for TTS. The benchmark contains 860 utterances annotated by trained linguist raters. That gives the paper a structured way to ask a sharper question: which automated evaluators can detect which kinds of speech issues?

They then benchmark four MOS predictors and four Audio-LLM judges against those annotations. The point is not to crown a single winner on a generic score, but to see whether these evaluators align with specific perceptual dimensions. That is a much more useful test if your goal is interpretable evaluation rather than just a single leaderboard number.

One important detail: the abstract does not provide per-metric benchmark numbers or a full score table. So while the paper clearly reports comparative findings, the source material here does not let us quote exact performance values. What it does make clear is the direction of the result: the evaluators behave differently depending on the kind of error and the way they are prompted.

What the paper actually shows

The headline result is that MOS predictors collapse onto acoustic signal quality. That means they seem to behave like broad sound-quality detectors rather than nuanced speech-error detectors. If you are using them as a proxy for “overall speech quality,” that may be fine for some workflows, but it is not the same as understanding which linguistic dimensions are broken.

Audio-LLM judges do somewhat better in a different way: they show selective, prompt-dependent detection. But that sensitivity does not generalize across all dimensions. In practice, that means prompt wording can change what the judge notices, and a model that catches one class of issue may miss another class entirely.

The paper’s broader conclusion is that neither class reliably captures a breadth of linguistically structured speech errors. That is the key limitation for practitioners. If you only use one automated evaluator, you may be measuring a narrow slice of the problem and calling it comprehensive evaluation.

Because the authors release their dataset, annotation schema, and evaluation code publicly, the work is also positioned as infrastructure, not just analysis. Even without benchmark numbers in the abstract, that release matters: it gives the community a shared testbed for probing whether future evaluators actually understand the dimensions that humans hear.

Why developers should care

If you work on TTS, this paper is a reminder that evaluation can shape product decisions. A model optimized to improve MOS may get better at sounding clean while still producing subtle linguistic mistakes. Those mistakes can hurt trust, accessibility, and downstream usability even when the audio “sounds natural.”

The paper also suggests that Audio-LLM judges are not a drop-in replacement for human raters. Prompting matters, and prompt dependence is a real source of instability. That means teams should be cautious about using a single judge prompt as a universal quality gate.

For engineering workflows, the most useful lesson is to evaluate at the level of failure mode, not just at the level of aggregate score. A dimension-aware benchmark can help you answer questions like: Is the system failing on a specific pronunciation pattern? Is it missing prosodic structure? Is the evaluator blind to that issue too?

Limitations and open questions

The abstract gives a clear direction, but it also leaves some practical questions open. We do not get the detailed dimension definitions in the source material here, so readers cannot yet judge exactly how broad or fine-grained the schema is from the abstract alone.

We also do not get exact benchmark scores, error bars, or per-dimension breakdowns in the provided notes. That means this summary can confidently report the qualitative findings, but not the full quantitative ranking of the four MOS predictors and four Audio-LLM judges.

Another open question is how well the benchmark transfers beyond the annotated set of 860 utterances. The dataset is a strong start, but broader coverage across languages, voices, and synthesis styles would be needed before anyone could treat the results as the final word on TTS evaluation.

Still, the contribution is clear: the paper moves evaluation away from a single vague notion of naturalness and toward a more inspectable, linguistically grounded view of speech quality. For teams shipping TTS systems, that shift is likely to matter as much as any single score improvement.

What to take away

• Naturalness alone is too coarse to evaluate TTS reliably.
• MOS predictors and Audio-LLM judges miss different kinds of speech errors.
• Dimension-level benchmarks can make TTS evaluation more interpretable and actionable.