AI translation of literary texts is "fine", but readers still prefer human translations
Yves Ferstler, Adam Podoxin, Ty Brassington, Roman Grundkiewicz, Maite Taboada, Marzena Karpinska
Read on arXiv →
Key claim
Readers prefer human translations for their clarity and immersion.
In plain English
Imagine you're a reader diving into a novel translated from another language. You want the experience to be immersive and engaging, but how do you know if a machine translation captures the essence of the original text? Currently, most evaluations focus on whether the translation is fluent or adequate, but they miss the emotional and literary impact that makes reading enjoyable. This is where things can go wrong: machine translations might seem fine on the surface, but they often lack the depth and clarity that human translators provide. This mismatch is what's called a failure of literary effect in machine translation.
This paper takes a fresh approach by asking avid readers to compare human translations to those generated by a large language model. They evaluated excerpts from recent novels in different languages, focusing on how immersive and clear each translation felt. The results showed that while machine translations were generally acceptable, readers preferred human translations for their ease and immersive quality. Interestingly, readers often struggled to tell the two apart, indicating that machine translations can sometimes be quite good, but they still fall short in capturing the full literary experience.
What’s new here is the introduction of a reader-centered evaluation dataset, which includes thousands of reader comments and judgments. This approach shifts the focus from traditional metrics to actual reader experiences, providing a more nuanced understanding of translation quality. For anyone building translation systems, this means that relying solely on automatic metrics might not be enough; understanding how readers perceive translations is crucial for improving the quality of machine-generated texts.
The paper introduces a reader-centered evaluation approach for machine translation, which is a significant extension of existing methods.
The study is based on a solid experimental design with multiple comparisons and reader evaluations.
Deep reliability assessment
The methodology supports the claim that, for 8K-word openings of recent FR/PL/JA novels translated into English, avid readers slightly prefer professional human translations and cannot reliably identify AI translations. It overclaims if generalized to full books, other genres, other language pairs, production workflows with human post-editing, or the broader reading public, since the study uses 15 readers, 15 books, and two readers per book.
Reproducibility
Yes. The authors say they release the LAIT dataset, evaluation protocol, supporting interface, reader comments, judgments, preference ratings, and span-level annotations at github.com/Yves575/lait, with a project site at http://lait.cs.sfu.ca/; copyright limits may mean only excerpts or partial texts are shared.
Key figure
Figure 1 shows the evaluation pipeline: each reader reads two full 8K-word translation versions in counterbalanced order, answers questionnaires, compares them, then after a break performs side-by-side chunk-level close reading with preferences, justifications, confidence, and span highlights.