Overview
AFP on a manipulation sequence. Values near 1 mark the objects and robot parts the task depends on.
Robotic foundation models still need task-specific fine-tuning before they work reliably in a new setting, and that fine-tuning supervises actions only. Nothing constrains which visual evidence the policy adapts to, so it can settle on cues that merely predict the demonstrations: background texture, lighting, object co-occurrence, camera artifacts. These shortcuts hold in the training scene and fail under distribution shift.
AFP is a small, policy-agnostic module that takes the same RGB frames and instruction as the policy and predicts a continuous mask over the task-relevant objects, the robot and other action-critical regions. During fine-tuning the mask supervises the policy's attention over image tokens. The architecture is unchanged, and once fine-tuning ends the policy runs on the raw observation stream without AFP. Across SmolVLA, OpenVLA, π0.5 and Motus on eight simulated tasks, and π0.5 on five real-robot tasks, this raises success under visual distractors, matches or improves in-distribution success on every task, and shortens fine-tuning on long-horizon tasks.
Method
Given a sequence of frames and a task description , AFP predicts one dense relevance mask per frame.
The instruction reweights the region queries through a FiLM adapter instead of being concatenated as tokens, and the text encoder stays frozen.
Per-channel scale in [−1, 1], shift , and gate in [0, 1], all from the CLIP embedding .
During fine-tuning we read the policy's attention over image tokens, average it over trainable layers, heads and query positions, and compare the result with the pooled AFP mask. Both sides are normalized, so the loss compares distributions, not attention mass. The policy is trained on .
is the averaged policy attention over image tokens and the AFP mask pooled to the same token grid.
The grounding signal should steer action learning, not compete with it. Following PCGrad, when the AFP gradient points against the action gradient we drop its conflicting component and update with . Without the projection the total loss falls at first, then spikes after an onset point a little past the halfway mark of training. With it, training stays smooth.
In action
Attention attribution during out-of-distribution rollouts, with distractors added to the scene. Left: direct fine-tuning. Right: fine-tuning with the AFP loss. Same model, same demonstrations, same training budget.
The distractors are unused 3D-printed furniture parts, socks that never appeared in training, or an opened cabinet door. Clips end when the episode ends, so the two sides differ in length.
Top-camera sequences with the predicted mask applied. The mask follows the objects being manipulated and the gripper, and drops the rest.
Results
We compare each policy's attention attribution with human-labeled task masks over 10 in-distribution and 10 out-of-distribution rollouts per task, averaged across the eight simulated tasks. Soft-IoU measures overlap. EMD measures how far the attention mass sits from the mask. Direct fine-tuning leaves attention on background and distractors for every backbone we tried.
Eight MimicGen tasks, 500 demonstrations and 20k fine-tuning steps each. The out-of-distribution setting keeps the objective and adds three to five unrelated RoboSuite objects with random materials to the tabletop.
π0.5 on an i2RT YAM Arm, five tasks, 300 demonstrations and 50k fine-tuning steps each, 30 rollouts per setting. Drawer, Lamp and Table Assembly come from FurnitureBench. AFP improves every task in both settings, with the largest gains out of distribution.
Drawer Assembly rollouts in the training scene and with spare 3D-printed parts from the same furniture family added as distractors.
AFP also shortens fine-tuning. On short tasks such as Square D1 and Coffee D1, both runs level off at about the same step and the gain is modest. On Kitchen D1, Coffee Preparation D1 and Three Piece Assembly D1, which have longer episodes and several interaction phases, the AFP-trained policy reaches high success well before the directly fine-tuned one. The gain seems to grow with the share of the observation stream that is incidental to the task.
AFP depends on task-relevance annotation: humans label masks, AFP learns from them, and only then supervises the policy. That makes the supervision explicit and reusable, but adds labeling cost and possible bias. AFP can fail when relevance is ambiguous, heavily occluded, or outside the distribution of the mask dataset. Two directions follow: scale the dataset with the annotation tool, or learn relevance from interaction, temporal consistency, counterfactual scene changes and policy failures.
Labeling a real-robot episode, played at 5x. Point prompts on the first frame, a dozen instance layers, then propagation through the sequence.
Data and annotation
AFP is trained as a supervised mask predictor. Its targets are graded relevance fields, not binary segmentations: values fade around object boundaries, contact regions and partial occlusions. Standard annotation tools assign each pixel to one object, so we built our own.
With this tool we labeled frames from Open-X Embodiment, DROID, MimicGen, LIBERO and our own scenes. Each training example is an RGB frame, an instruction and a continuous mask. The training dataset is on Hugging Face.
Citation
@article{sun2026artificial,
title={Artificial Foveated Perception for Mitigating Shortcut Learning in Robotic Foundation Models},
author={Sun, Xiatao and Zhuang, Yuan and Negrete, Mateo Sanchez Lopez and Coldea, Matei-Victor and Liang, Chen and Zhang, Haoyang and Liu, Che and Zeng, Ziyao and Li, Shawn and Wang, Qian and others},
journal={arXiv preprint arXiv:2607.10655},
year={2026}
}