WinDOM: Self-Family Distillation for Small-Model GUI Grounding
Chengheng Li-Chen, Zhiqian Zhou, Hao Chen, Nicolas Chauvin
Read on arXiv →
Key claim
New training method improves small GUI-grounding agent performance.
In plain English
Imagine you're trying to build a small AI that can understand and interact with graphical user interfaces (GUIs) on devices like smartphones or tablets. The challenge is that creating training data for these models usually requires a lot of human effort to annotate images with bounding boxes, which can be costly and time-consuming. This is especially problematic for smaller models that are meant to run on devices with limited resources. When you try to scale down the model, you often lose performance because you don't have enough high-quality training data. This is what's called the data annotation bottleneck.
Currently, many developers either rely on expensive human annotation or use existing datasets that may not fit their specific needs. This can lead to models that perform poorly in real-world scenarios, especially when they encounter new types of interfaces or layouts that weren't part of their training data. This is known as the generalization problem, where the model struggles to adapt to unseen situations.
The authors of this paper propose a clever solution to these issues. They introduce a method for automatically generating a large dataset of GUI grounding data by simulating interactions with a web browser. Instead of relying on human annotators, they use a tool that can read the structure of web pages directly, creating bounding boxes around elements without any manual input. This approach not only saves time and resources but also allows for the creation of a diverse dataset that can help the model learn better.
Additionally, they present a new training technique called Self-Family Distillation (SFD), which helps improve the model's performance by using a smaller version of itself or a larger related model as a teacher. This method allows the smaller model to learn effectively from its own outputs, leading to better initializations for training. The results show that their approach leads to significant improvements in performance on various benchmarks, making it a practical option for developers looking to deploy small, efficient models for GUI tasks.
The paper introduces a novel approach to training small GUI-grounding agents using a unique data collection method and a new distillation technique.
The results are supported by a substantial dataset and clear performance improvements over baseline models.
Deep reliability assessment
The paper convincingly shows that a DOM-derived dataset plus an under-saturated SFD cold-start seeding Early-init GRPO improves OOD click-in-target accuracy over a Qwen3.5-2B base across three benchmarks with deterministic rewards and greedy decoding. Claims about broader real-world safety and generalization beyond the Win11React-rendered data and screenshot benchmarks are not directly supported by end-to-end agent evaluations or comparisons against stronger external baselines.
Reproducibility
Partial: the paper claims to release WinDOM with a deterministic regeneration script and uses a deterministic point-in-box reward and bit-deterministic greedy decoding; however, no code/checkpoint URL is provided in the excerpt, and full training details are deferred to appendices.
Key figure
A pipeline diagram where Win11React renders screens to produce DOM-derived bounding boxes, feeding Self-Family Distillation (EMA or 4B teacher) to create an under-saturated checkpoint that seeds Early-init GRPO with a deterministic point-in-box reward.