Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs

Imagine you're trying to teach a robot how to perform tasks based on visual inputs and instructions. The challenge is that gathering expert demonstrations — the specific actions taken in response to various observations and instructions — is expensive and time-consuming. This often leads to a bottleneck where you can't scale your training effectively because you don't have enough of these expert examples. When you do manage to collect some, the robot might still struggle because it needs to learn not just how to move but also what those movements mean in context. This is where things can go wrong: if the robot focuses too much on the physical actions without understanding the instructions, it won't perform well in real-world scenarios. This is what's called a conflation of learning objectives, where the robot's physical competence and semantic understanding are not aligned properly. The authors propose a solution to this problem by introducing a two-stage training framework called Task-Agnostic Pretraining (TAP). In the first stage, the robot learns from a wide range of unlabeled interactions, including off-task movements and playful actions, which helps it develop a sense of how to move. Then, in the second stage, it grounds this knowledge in language using minimal expert data. This approach allows the robot to build robust physical representations without needing vast amounts of labeled data. The results are promising: on a benchmark, TAP matches the performance of models trained on over a million expert trajectories while using far less labeled data. In practical terms, this means that if you're building an embodied AI system, you can achieve better performance with less reliance on costly expert demonstrations, making your development process more efficient.

The paper introduces a new framework that effectively separates learning objectives in VLA models, which is a significant advancement.

The results are supported by strong benchmarks and demonstrate clear improvements over existing methods.