When a scene suggests only one task, a robot can succeed while barely using language. RoboFollow asks what happens when the same scene supports different intentions.
Instruction A selects the lower-right yellow block. Illustrative target highlighting on a scene from the paper.
4diagnostic scene families
75training task labels
3,750training demonstrations
9VLA & WAM policies
01 / THE IDEA
Make language necessary.
High task success does not necessarily imply instruction following.
RoboFollow is a diagnostic benchmark built on RoboTwin. Each training scene supports multiple feasible task branches, so the observation alone cannot identify the intended behavior. A four-level protocol isolates visual grounding, semantic recombination, and their combination. Stage-wise Intent and Execution scores distinguish task-selection errors from motor failures.
Across nine fine-tuned VLA and WAM policies, strong in-distribution performance, where attained, does not reliably transfer to the perturbed settings. Stronger VLM backbones, QA co-training, LangForce, and classifier-free guidance do not close this gap in the evaluated configurations.
Hscene = Htrain(T | S)
3.782 bits in RoboFollow vs.0.880 bits in the equally weighted LIBERO Spatial/Object/Goal/Long suites. Scene entropy is a dataset design measure; see the paper for task grouping.
02 / THE BENCHMARK
Four ways to test what language means.
Simple objects. Familiar motor primitives. Multiple possible intentions.
Which object does the relation identify?
Identical yellow blocks must be distinguished by their relation to anchor objects. Moving the anchors tests grounding beyond memorized coordinates.
Spatial prepositions · 16 training tasks
Spatial grounding under controlled layout and instruction changes. Paper, Figure 2. Open full-size figure ↗
Can the robot compose object attributes?
Color, size, shape, and action must be bound together. Shared attributes make any single visual cue insufficient across tasks.
L2Semantic recombinationFamiliar layout New composition
L3Joint generalizationChanged layout New composition
The levels isolate different generalization dimensions; they are not a strict difficulty ranking.
STAGE-WISE / INTENT & EXECUTION
Did it choose correctly? Did it execute successfully?
IS
Intent Score
The correct object, relation, waypoint, orientation, or logical branch was selected.
ES
Execution Score
The corresponding physical subgoal was completed successfully.
CR
Completion Rate
An action-dependent binary completion criterion, averaged equally over tasks within each scene and level.
A finishing stage contributes 20% of IS/ES and checks that the robot stops unrelated actions after completing the instruction.
03 / THE EVIDENCE
The gap appears beyond the familiar.
Nine policies. Four scene families. Explore the results from Table 1.
Selected policyOther evaluated policiesπ₀.₅ empty-language control
All values are percentages. The dashed control is the π₀.₅ empty-language condition. Results characterize the evaluated fine-tuning setup; they do not isolate architecture from training data or optimization.
View the numbers All nine policies + language ablation
Intent Score (%) · unweighted mean over four scenes
Understanding a scene does not guarantee following an instruction.
01
Visual shortcuts break.
In Scene 1, π₀ reaches 98.2% IS at L0 but 0.0% at L1 after layout changes. Task success on a familiar scene can conceal fragile spatial grounding.
02
Better QA is insufficient.
Qwen3-VL-4B answers 19/20 Scene 2 QA probes correctly. Yet the tested stronger-backbone and QA co-training setup still leaves substantial L1–L3 action-grounding failures.
03
The tested fixes leave a gap.
LangForce yields marginal changes; classifier-free guidance at scales 1.2 and 1.5 degrades even L0. These mitigations do not resolve generalization in the tested configurations.
Inspect representative failure cases Paper · Figure 7Failure cases reproduced from the paper. Select the figure to view it at full size.
Real-robot pilot · π₀.₅ · paper, Figure 6
A PRELIMINARY REAL-WORLD CHECK
From training instructions to held-out compositions.
50%Training instructions · 20/40
→
15%Held-out instructions · 6/40
Eight instructions per set, five trials each, using the same object set. This small pilot compares different instruction sets and pick/stack compositions; it is not a matched-pair study.
@misc{guo2026robofollow,
title = {RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents},
author = {Guo, Chang and Xie, Yukun and Tan, Bohan and Chang, Zheng and Yin, Zhaokai and Ma, Qianli and Wang, Yingqiao and Liang, Chao and Zhang, Zhipeng},
year = {2026},
eprint = {2609.25636},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2609.25636}
}
RoboFollow
