TriViewBench: Controlled Complexity Scaling for Multi-View Structural Reasoning in MLLMs

Imagine you're building a system that needs to answer questions about images, like identifying objects or understanding scenes. Current models can handle this to some extent, but they struggle when the tasks get more complex, especially when it comes to counting objects or understanding the overall scene. This is because they often miss important details or get confused by overlapping objects. For instance, they might undercount items that are partially hidden or overcount when multiple views create confusion about what’s what. This is known as occlusion blindness and cross-view identity confusion, respectively.

To tackle these issues, the authors created TriViewBench, a new benchmark that tests how well these models can reason about visual information in a controlled way. They constructed a set of synthetic 3D scenes with varying complexity and asked models to answer questions about them. What they found was that all the models they tested performed similarly, with a clear hierarchy in their capabilities: they were much better at simple tasks than at more complex ones. For example, when asked to count objects, the models showed a significant drop in accuracy as the task complexity increased, with some tasks leading to an 80% drop in performance.

This work is important because it not only highlights the limitations of current multimodal models but also provides a structured way to analyze where they fail. By using TriViewBench, builders can better understand the specific weaknesses in their models and work on improving them, especially in areas where reasoning about complex visual information is required.

The introduction of TriViewBench provides a new benchmark for evaluating visual reasoning in MLLMs, addressing scalability issues.

The evaluation of 18 models under a unified protocol and detailed error analysis supports the claims made.