How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations
Yuxing Cheng, Yuan Wu, Yi Chang
Read on arXiv →
Key claim
Higher accuracy on clean data doesn't ensure robustness.
In plain English
Imagine you're building a system that relies on reading text from images, like receipts or documents. You want it to be accurate, but what happens when the images are blurry or distorted? In real life, visual corruption can lead to errors in reading text, which can mess up the entire reasoning process. This is a significant issue because if your model can't handle these imperfections, it might give wrong answers based on faulty data. This problem is known as robustness under visual degradation, and it's a critical area for improving OCR systems.
Currently, many models are evaluated based on their performance on clean images, but this doesn't tell the whole story. When faced with real-world conditions, these models often struggle. For instance, they might perform well on clear documents but fail miserably when dealing with charts or handwritten notes that are slightly distorted. This is what's called a lack of robustness, and it can lead to significant errors in applications that rely on accurate text recognition.
To address this, the authors created a new benchmark called OCR-Robust, which systematically evaluates how well models can handle visual perturbations. They tested 18 different models across various types of text inputs, from documents to charts, under different levels of visual degradation. The findings reveal that just because a model has high accuracy on clean data doesn't mean it will be reliable in the real world. In fact, some models showed a dramatic drop in performance when faced with challenging inputs like charts and tables. This insight is crucial for anyone building OCR systems, as it emphasizes the need to consider robustness in addition to accuracy when developing and deploying these models.
The introduction of the OCR-Robust benchmark provides a new way to evaluate OCR reasoning under visual degradation.
The study uses a solid evaluation framework with multiple models and perturbations, supporting its claims.
Deep reliability assessment
The methodology supports a controlled diagnostic comparison of OCR-reasoning robustness across models under five selected synthetic perturbations. It does not fully support claims about real-world deployment robustness, because the dataset is moderate-sized, perturbations are mostly single-factor, the pilot selection basis is narrow, and prompting/OCR pipeline coverage is limited.
Reproducibility
Yes: the paper states that code and supplementary materials are available at the GitHub repository, but the excerpt only explicitly confirms code/supplementary release rather than clearly specifying full dataset licensing or hosting details.
Key figure
The key diagram likely illustrates the OCR-Robust evaluation pipeline: clean OCR-reasoning samples are transformed with selected visual perturbations at multiple severities, then evaluated across VLMs and OCR+LLM pipelines using clean accuracy, RCR, WCR, and CRI.