Fast-VLA: You Only Think Once for Robot Manipulation

Abstract

Vision–language–action (VLA) policies commonly place a multi-billion-parameter model inside the feedback loop, repeatedly recomputing task context whenever they predict a new action chunk. Yet task understanding changes far more slowly than the robot state. We introduce Fast-VLA, which invokes a frozen 4B VLM once per episode to contextualize the instruction and initial views as fixed, multi-layer key–value task memory. During execution, a lightweight online actor fuses learned time- and arm-indexed queries with current visual and proprioceptive feedback to form action queries. These queries retrieve task context from the fixed memory and predict action chunks without re-running the foundation model.

Fast-VLA achieves 92.46% and 92.40% success on RoboTwin 2.0 Clean and Randomized, and 95.95% and 79.11% on LIBERO and LIBERO-Plus evaluation. Against three continually conditioned baselines on the same hardware, it reduces RoboTwin amortized policy inference time by 10.9–17.1× and requires 1.22 GiB of post-prefill online resident memory. Task-memory swaps further show that the actor relies on episode-matched task memory. These results show that task context can be established once while closed-loop control remains responsive throughout execution.

Fast-VLA overview — One prefill per episode, then the online actor runs closed-loop by itself.

Results

Task success and inference cost on RoboTwin 2.0, LIBERO, and LIBERO-Plus. Latency and memory are measured on identical hardware for all methods. Amortized inference time includes episode initialization.

Type	Method	Action PT	Pred./exec.	ms/chunk ↓	ms/step ↓	GiB ↓	Clean ↑	Rand. ↑	Mean ↑
VLA	π₀	✓	50/50	72.38	1.580	6.26	65.92♭	58.40♭	62.16
VLA	π₀.₅	✓	50/50	80.51	1.758	6.47	82.74♭	76.76♭	79.75
VLA	X-VLA	--	30/30	211.03	7.452	3.29	72.90♭	72.80♭	72.85
VLA	LingBot-VLA	✓	50/25	740.35	30.919	7.99	86.50	85.34	85.92
VLA	Qwen-VLA	✓	--	--	--	--	86.10	87.20	86.65
VLA	LingBot-VLA 2.0	✓	50/25	931.29	38.892	11.98	93.52	92.80	93.16
VLA	DM0.5	✓	50/50	736.10	16.069	11.68	93.60	93.30	93.45
WAM	Motus	✓	16/16	2378.24	152.965	28.17	88.66♭	87.02♭	87.84
WAM	LingBot-VA	✓	32/32	9553.12	317.765	29.59	92.90♭	91.50♭	92.20
WAM	Fast-WAM	✗	32/24	564.80	24.624	23.13	91.88	91.78	91.83
WAM	Flash-WAM	✗	32/32	985.10	32.767	26.26	88.42	82.66	85.54
WAM	AHA-WAM	✗	16/16†	553.27†	39.291	28.21	93.40	92.20	92.80
WAM	Faster-WAM (Noah)♯	✗	32/24	119.94	5.229	21.34	89.70	88.64	89.17
WAM	Faster-WAM (HUST)	✗	32/28	416.12	15.704	22.27	92.78	92.26	92.52
Ours	Fast-VLA	✗	50/50	44.87¶	1.437¶	1.22¶	92.46	92.40	92.43

Fast-VLA reaches 92.46% Clean and 92.40% Randomized success (92.43% mean), with 10.9×, 11.2×, and 17.1× lower amortized inference time than Faster-WAM (HUST), DM0.5, and Fast-WAM.

† AHA-WAM performs a 111.40 ms video prefill every two chunks, included in ms/step. ¶ Fast-VLA takes 153.74 ms for its first chunk (108.87 ms prefill + 44.87 ms actor), then 44.87 ms per chunk. Its online resident memory is 1.22 GiB; peak memory during prefill is 12.11 GiB.

Type	Method	Action PT	Pred./exec.	ms/chunk ↓	Amort. ms/step ↓	GiB ↓	LIBERO-Std ↑	LIBERO-Plus ↑
VLA	π₀	✓	50/5	81.96	16.577	6.26	94.10	53.60
VLA	π₀-FAST	✓	10/5	353.06	71.885	5.57	85.50♭	61.60
VLA	OpenVLA-OFT	✓	8/8	107.00	13.685	14.41	97.10♭	69.60
VLA	π₀.₅	✓	10/5	86.35	17.466	6.47	96.90	85.70
VLA	UniVLA	✓	12/1	241.96	241.960	14.15	95.50♭	43.90
VLA	X-VLA	--	30/30	199.40	7.220	3.29	98.10	72.56‖
VLA	HyperVLA†	✓	4/1	9.59	16.385	1.71	89.00	--
VLA	DM0.5	✓	10/10	701.56	71.780	11.58	98.50‖	85.90‖
WAM	Motus	✓	--	--	--	--	97.70	--
WAM	LingBot-VA	✓	16/16	6913.53	450.416	22.96	98.50	--
WAM	Fast-WAM	✗	32/10	544.50	55.710	23.13	97.60	49.14
WAM	Joint-WAM	✗	32/10	916.81	93.802	23.13	--	66.27
WAM	Faster-WAM (Noah)♯	✗	32/10	118.85	12.160	21.35	98.50	75.00
WAM	Faster-WAM (HUST)	✗	32/10	824.43	84.350	22.27	99.00	73.57
Ours	Fast-VLA	✗	10/10	34.25¶	3.975¶	0.86¶	95.95	79.11

Fast-VLA reaches 95.95% on standard LIBERO and 79.11% on LIBERO-Plus, with 3.975 ms amortized inference time per environment step.

† HyperVLA success is from its original paper, with OXE pretraining and LIBERO fine-tuning; inference cost is measured using our implementation. Its 1168.01 ms initialization and 9.59 ms per-step policy call yield 16.385 ms/step. ¶ Fast-VLA takes 122.63 ms for its first chunk (88.38 ms prefill + 34.25 ms actor), then 34.25 ms per chunk. Online resident memory is 0.86 GiB; peak memory during prefill is 11.74 GiB.

Pred./exec. denotes actions predicted per call / executed before replanning. Action PT denotes external action or latent-action pretraining of action-generating weights; VLM or visual-backbone pretraining alone does not count. GiB denotes online resident memory. ♭ Benchmark compilation; ‖ locally evaluated released model; ♯ local implementation used for inference-cost profiling. Baseline success rates are drawn from published reports, benchmark compilations, or our evaluations of released checkpoints; detailed sources are provided in the paper. A dash denotes unavailable data.

Rollouts

Selected successful simulation rollouts. Every episode below runs on a single task memory, built before the first frame and never refreshed. Pick a clip from the thumbnails, or step through with the arrows. Use the video controls to pause or seek.

Videos are played at their encoded frame rates: 10 fps for RoboTwin 2.0 and 20 fps for LIBERO and LIBERO-Plus. Playback timing is for visualization and does not measure inference latency; see the Results section for measured inference costs.

RoboTwin 2.0

Bimanual tasks in the Clean setting. The third-person view is shown at full size with both wrist views inset; each actor call predicts a 50-step action chunk. Labels give the task step budget and the maximum number of actor calls allowed by that budget; individual rollouts may finish earlier.

LIBERO-10

Long-horizon single-arm tasks, third-person view with the wrist view inset.

LIBERO-Plus

The same task — put the black bowl in the bottom drawer of the cabinet and close it — under six LIBERO-Plus perturbations: camera viewpoint, background texture, lighting, object layout, sensor noise, and robot initial state. Task memory is built once from the perturbed initial observation and remains fixed for the episode. These selected examples illustrate behavior on one task; aggregate LIBERO-Plus success rates are reported in the Results section.

Where the actor looks

Actor attention during a successful stack_blocks_two episode.

Attention over a full episode — A successful 300-step stack_blocks_two rollout, shown in successive 50-step action chunks. As control passes from the left arm to the right, head-camera attention shifts from the red block to the green block. Each wrist focuses on its arm’s target; the right wrist attends to the green block before the second sub-goal begins. Crosshairs mark attention peaks; enrichment above 1 indicates preferential attention to the corresponding block. The final open-loop chunk has no new attention map.

Ablations and analysis

Task-memory dependence is evaluated on RoboTwin 2.0 (Clean and Randomized). The remaining panels report LIBERO and LIBERO-Plus results. All success rates are in %.

On RoboTwin 2.0, we alter the task-memory content at test time while keeping the actor weights fixed, without any additional training.

Memory condition	Clean	Rand.
Swapped language segment	35.18	29.66
Same task, different episode	6.26	8.36
Swapped initial-view segment	2.64	2.62
Different task	0.62	0.52
Original memory	92.46	92.40

Replacing task memory reduces success across all conditions. Memory from another episode of the same task reaches 6.26% Clean and 8.36% Randomized success, showing the importance of episode-specific context.

Task-memory construction on LIBERO (%). The actor is retrained separately for each construction variant.

Condition	Memory	Std. ↑	Plus ↑
No task memory	Constant zero KV	69.90	53.69
Text-token KV only	Image-conditioned text KV	93.20	73.84
Last layer only	Final-layer instruction + views KV	94.70	73.70
Full (Ours)	34-layer instruction + views KV	95.95	79.11

Online-input ablations on LIBERO (%). Each variant is retrained with the indicated combination of online inputs.

Condition	Head	Wrist	State	Std. ↑	Plus ↑
No head view	✗	✓	✓	92.70	74.54
No wrist vision	✓	✗	✓	65.60	33.49
No state	✓	✓	✗	94.30	72.41
Head only	✓	✗	✗	70.80	30.80
Wrist only	✗	✓	✗	87.10	68.72
Full (Ours)	✓	✓	✓	95.95	79.11

Task-memory refresh ablation on LIBERO using last-layer memory and matched training for each schedule. Amortized cost includes a 34.25-ms actor call per chunk and one VLM prefill per memory construction.

Memory schedule	VLM calls/episode ↓	Amort. ms/step ↓	Std. ↑	Plus ↑
Every action chunk	17.37	12.391	96.50	72.47
Every 2 chunks	8.93	8.052	95.95	70.63
Every 4 chunks	4.74	5.898	95.95	68.24
Once per episode	1.00	3.975	94.70	73.70

Task-memory encoders on LIBERO (%). The actor architecture is unchanged across rows; memory denotes the bf16 KV cache stored per episode in the LIBERO configuration.

Memory producer	Attention	Memory↓	Std. ↑	Plus ↑
PaliGemma v1	18L, MQA	9.8 MB	95.35	73.15
π₀.₅	18L, MQA	9.8 MB	94.90	71.48
Qwen2.5-VL-7B	28L, GQA-4	31.8 MB	95.35	75.55
DM0.5 (default)	34L, GQA-4	79.7 MB	95.95	79.11

Performance across task horizons

We evaluate fixed task memory across all 50 RoboTwin tasks, grouped by their official step budgets.

RoboTwin 2.0 success (%) by step budget: short (S), 400–500 steps (29 tasks); medium (M), 501–900 (15 tasks); long (L), 901–1700 (6 tasks).

Type	Method	Short	Medium	Long	All 50
VLA	π₀	61.52	68.43	49.58	62.16
VLA	π₀.₅	81.31	80.60	70.08	79.75
VLA	X-VLA	76.41	68.93	65.50	72.86
VLA	LingBot-VLA	85.29	87.70	84.50	85.92
WAM	Motus	89.09	85.93	86.00	87.77
WAM	Fast-WAM	93.48	90.53	83.67	91.42
Ours	Fast-VLA	94.00	90.27	90.25	92.43

Fast-VLA reaches 94.00%, 90.27%, and 90.25% success on short-, medium-, and long-horizon tasks. On put_bottles_dustbin, it reaches 87.00% under a 1,700-step budget, permitting up to 34 actor calls with one task memory.

Per-task results for π₀, π₀.₅, X-VLA, Motus, and LingBot-VLA come from the StarVLA compilation, using RGB-only LingBot-VLA results. Fast-WAM results come from our evaluation of its released model. These per-task baseline results are sourced separately from the aggregate results above, so their overall averages may differ. Group success rates pool Clean and Randomized outcomes and are computed as total successes divided by total episodes. For Fast-VLA, we evaluate 100 episodes per task in each setting (10,000 episodes in total); the All 50 column pools all tasks and both settings.