Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning
Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
Read on arXiv →
Key claim
Smaller models can outperform larger ones in task planning.
In plain English
Imagine you're trying to build a digital assistant that can help you with repetitive tasks across different websites. The challenge is that these tasks can be complex and require careful planning to break them down into manageable steps. Currently, many models struggle with this because they either lack the ability to plan effectively or don't generalize well across different environments. This is particularly problematic when the tasks vary significantly from what the model was trained on, leading to failures in execution — a situation known as poor out-of-distribution (OOD) generalization.
To tackle these issues, the authors propose a method that allows the model to explore different environments and learn from past experiences. This approach helps the model to create high-level training data that is closely aligned with the tasks it needs to perform. They also introduce a framework to analyze how well the model can generalize across different levels of task complexity, revealing that just mastering simple tasks doesn't guarantee success in more complex planning scenarios.
What sets this work apart is that their smaller model, with only 7 billion parameters, achieves a 30.6% accuracy rate, outperforming a much larger model with 32 billion parameters. This suggests that with the right training and experience utilization, smaller models can be just as effective, if not more so, than their larger counterparts in certain applications. For builders, this means that investing in efficient training methods for smaller models could yield significant benefits in real-world task execution.
The paper introduces a new method for task planning in multimodal agents, addressing limitations of existing models.
The claims are supported by experiments on real-world benchmarks, demonstrating improved performance over larger models.
Deep reliability assessment
The methodology supports the claim that hindsight relabeling of autonomous web exploration traces can improve cross-website planning accuracy for small MLLM GUI agents on information-seeking and navigation tasks. It overclaims if read as evidence for general GUI automation, since the evaluation excludes authenticated sessions, CAPTCHA, payments, and other high-risk stateful workflows.
Reproducibility
No code repository is mentioned in the provided paper text. The evaluation uses the public WebVoyager benchmark, but the autonomously explored training trajectories, generated PEEU tasks, and full prompts are only described as being in appendices, not clearly released.
Key figure
Figure 1 contrasts prior atomic or coarse task-to-trajectory training with PEEU’s loop of autonomous exploration, trajectory-to-experience extraction, and hindsight generation of better-aligned high-level tasks.