Hallucination in World Models is Predictable and Preventable

Imagine you're building a system that predicts future scenarios based on current actions, like a video game AI that needs to plan its moves. The challenge is that these systems often generate plausible-looking outputs that don't match reality — this is called hallucination. It happens especially in areas where the model hasn't seen enough examples, leading to unreliable predictions. For instance, if the AI has never encountered a specific type of terrain, it might create a convincing but incorrect representation of how to navigate it. This is what's known as low-coverage regions in the state-action space, where the model's understanding is weak.

To tackle this problem, the authors introduce a new dataset called MMBench2, which includes a vast array of tasks and ground-truth actions. They also identify three specific types of hallucination that occur at different stages of the model's processing pipeline. By developing signals that can predict when and where these hallucinations will happen, they create a method to improve the model's training and performance. This involves using these signals to guide data collection and fine-tuning, allowing the model to adapt to new environments with minimal real-world data.

Overall, this approach shifts the focus from merely generating outputs to ensuring those outputs are grounded in reality, significantly enhancing the reliability of generative models in practical applications.

The introduction of MMBench2 and the identification of distinct hallucination modes represent a significant advancement in understanding and mitigating hallucination in generative models.

The paper provides a solid experimental foundation with a large dataset and clear methodologies for addressing hallucination issues.