Fast-VLA: You Only… | arXiv Physical AI Research Summary

1. Key Themes

Separation of Task Understanding from Closed-Loop Control

The paper's central architectural insight is that task understanding (what am I doing, what objects matter) changes far more slowly than robot state (where is my arm, what do I see right now). Fast-VLA runs a frozen 4B VLM exactly once per episode to produce a fixed key-value "task memory," then a lightweight online actor (272M parameters for bimanual, 184.7M for single-arm) handles all subsequent control decisions by querying that memory with live observations. As stated in Section 1: "This motivates a policy that establishes task context once and then stays responsive to live observations throughout execution." The actor predicts 50-step action chunks in a single forward pass without re-running the VLM, achieving 92.43% mean success on RoboTwin 2.0 and 95.95% on standard LIBERO (Tables 2 and 3).

10.9–17.1× Inference Speedup at Competitive Accuracy

Against three continually conditioned baselines measured on identical hardware, Fast-VLA reduces amortized inference time by 10.9–17.1× on RoboTwin 2.0. Specifically, Table 2 shows Fast-VLA at 1.437 ms/step versus DM0.5 at 16.069 ms/step (11.2× faster), Faster-WAM (HUST) at 15.704 ms/step (10.9× faster), and Fast-WAM at 24.624 ms/step (17.1× faster). The success rate gap is negligible: Fast-VLA scores 92.43% mean versus DM0.5's 93.45% — a 1.02 percentage point difference. For deployment, this means a robot can run at control frequencies that are practical for real manipulation without sacrificing task competence.

Task Memory Is Actively Used, Not Decorative

The paper includes a critical validation that the online actor genuinely depends on episode-matched task memory. Table 4 shows that swapping the language segment drops success to 35.18%/29.66%, swapping initial-view segments collapses it to 2.64%/2.62%, and using memory from a different episode of the same task yields only 6.26%/8.36%. Memory from a completely different task drops to 0.62%/0.52%. This is strong evidence that the architecture isn't just ignoring the VLM output — the actor is actively retrieving task-specific context from the frozen KV cache at every control decision.

Memory Refresh Frequency: Once Is Enough

Table 8 directly tests whether refreshing task memory mid-episode helps. Refreshing every action chunk (17.37× more VLM calls) improves standard LIBERO by only 1.80 percentage points while reducing LIBERO-Plus robustness by 1.23 points and tripling amortized inference time. Refreshing every 2 or 4 chunks actually hurts LIBERO-Plus by 3.07 and 5.46 points respectively. The paper concludes: "These results support once-per-episode memory construction as an efficient trade-off between success rate and VLM computation."

Cross-Encoder Portability of Task Memory

Table 9 shows that task memory constructed by four different frozen VLMs (PaliGemma v1, π0.5, Qwen2.5-VL-7B, DM0.5) all produce usable task memory with the same actor architecture. Standard LIBERO success spans only 94.90–95.95% across encoders, but robustness (LIBERO-Plus) varies more significantly: DM0.5 leads at 79.11% versus PaliGemma's 73.15%. This suggests the actor architecture is the primary driver of base performance, while VLM choice matters most for robustness under perturbation.

2. Contrarian Perspectives

You Don't Need the VLM in the Control Loop

The dominant architecture pattern in VLA models (π0, π0.5, OpenVLA, X-VLA, and most WAM variants) places the full VLM backbone in the repeated control path, re-encoding images and instructions at every action prediction. Fast-VLA challenges this directly: "Must the large VLM backbone also be re-run at every control decision?" (Section 1). The evidence shows that removing the VLM from the loop after one prefill preserves competitive success while delivering order-of-magnitude speedups. Most robotics companies building VLA systems would assume the VLM must process every new observation — this paper provides empirical evidence that task context, once established, can be cached and queried cheaply.

Fixed Initial-View Memory Survives Scene Changes

A natural objection is that a fixed memory built from initial observations cannot represent a changing world. The paper addresses this by showing that the online actor's live visual encoders handle state estimation, while the fixed memory provides task context. Figure 4 demonstrates that attention shifts dynamically during execution — as control passes from the left arm to the right arm, head-camera attention shifts from the red block to the green block, and each wrist camera focuses on its arm's target object. The memory is fixed, but the queries into it change with every control step. On long-horizon tasks (901–1700 steps), Fast-VLA still achieves 90.25% success (Table 7), including 87.00% on a 1,700-step task requiring 34 actor calls from the same memory.

Wrist Vision Is More Important Than Head Vision for Closed-Loop Control

Table 6 reveals a surprising finding: a wrist-only policy (no head view, no proprioception) achieves 87.10% on standard LIBERO and 68.72% on LIBERO-Plus, while a head-only policy (no wrist vision, no proprioception) reaches only 70.80% and 30.80%. Removing wrist vision from the full model drops success to 65.60%/33.49%, while removing proprioception only reduces it to 94.30%/72.41. The paper states: "These results identify wrist vision as a key source of closed-loop feedback." This challenges the common assumption that third-person (head) views are the primary visual input for manipulation policies.

3. Companies Identified

Huawei Technologies Co., Ltd / Huawei Central Research Institute

Description: The authors' institution; both the paper and the Fast-VLA system are Huawei products.
Why relevant: Huawei is investing in Physical AI foundation models. Fast-VLA demonstrates an efficient VLA architecture that could be deployed on edge devices with limited compute, aligning with Huawei's hardware-software integration strategy.
Quote: "1Huawei Technologies Co., Ltd, 2Huawei Central Research Institute" (author affiliations)

Physical Intelligence (π0, π0.5)

Description: VLA model company; π0 and π0.5 are leading generalist robot policies.
Why relevant: π0.5 is a direct baseline, achieving 79.75% mean on RoboTwin and 85.70% on LIBERO-Plus. Fast-VLA trails π0.5 on LIBERO-Plus by 6.59 points but is 12.2× faster on RoboTwin amortized inference. π0.5 uses pretrained action-generating weights, which Fast-VLA does not.
Quote: Table 2 and Table 3 show π0.5 results; "Both baselines use pretrained action-generating weights, whereas our action decoder and head are trained from scratch on LIBERO demonstrations" (Section 4.2)

Dexmal Team (DM0.5)

Description: Producer of the DM0.5 4B VLM used as the frozen backbone for Fast-VLA's task memory construction.
Why relevant: DM0.5 is both the VLM backbone Fast-VLA uses and a competitive baseline. As a baseline, DM0.5 achieves 93.45% mean on RoboTwin (highest in the table) but at 16.069 ms/step. As Fast-VLA's backbone, it produces the best LIBERO-Plus results (79.11%) among tested VLMs.
Quote: "A frozen DM0.5 (Dexmal Team, 2026) 4B VLM" (Section 3.2); Table 9 shows DM0.5 as the best memory encoder

Google DeepMind (PaliGemma)

Description: Producer of PaliGemma v1, a 3B VLM tested as an alternative task memory encoder.
Why relevant: PaliGemma offers a compact alternative with 9.8 MB memory cache versus DM0.5's 79.7 MB, but at the cost of 5.96 percentage points lower LIBERO-Plus success. This trade-off matters for edge deployment decisions.
Quote: Table 9 shows PaliGemma results; "PaliGemma offers a compact alternative with a 9.8-MB cache versus DM0.5's 79.7 MB" (Section 4.4)

Alibaba (Qwen2.5-VL-7B)

Description: Producer of Qwen2.5-VL-7B, tested as an alternative task memory encoder.
Why relevant: Demonstrates that Fast-VLA's architecture is VLM-agnostic and can leverage different foundation models. Qwen2.5-VL-7B achieves 75.55% on LIBERO-Plus with a 31.8 MB cache.
Quote: Table 9 shows Qwen2.5-VL-7B results

4. People Identified

Kehong Gong

Lab/Institution: Huawei Technologies Co., Ltd
Why notable: Lead author of Fast-VLA; likely the primary architect of the online actor and query fusion mechanism.
Quote: Listed as first author; corresponding work on the full system architecture.

Michael Bi Mi

Lab/Institution: Huawei Technologies Co., Ltd
Why notable: Senior author at Huawei, likely leading the Physical AI research program. The combination of efficiency-focused architecture and competitive benchmarks suggests a deployment-oriented research agenda.
Quote: Listed as senior author (last position).

Mingyuan Zhang

Lab/Institution: Huawei Central Research Institute
Why notable: Corresponding author, indicating primary intellectual direction for the research. Based at Huawei's central research arm, suggesting this work is part of a broader Physical AI strategy.
Quote: "∗Corresponding author" (author footnote)

5. Operating Insights

VLM Compute Cost Can Be Amortized Across an Entire Episode

For teams deploying VLA models, the most actionable finding is that VLM inference need not be in the critical control path. Fast-VLA's architecture — one VLM prefill (~109 ms for RoboTwin) then 44.87 ms per action chunk — means the expensive foundation model call happens once, and subsequent control decisions cost less than a lightweight transformer forward pass. For a 237-step RoboTwin episode, the amortized cost is 1.437 ms/step. This has direct implications for edge deployment: you need enough memory for the VLM prefill (12.11 GiB peak) but only 1.22 GiB resident during execution. A CTO evaluating VLA deployment should ask whether their architecture truly needs the VLM at every step or whether a cached task representation would suffice.

Action Decoders Trained from Scratch Can Compete with Pretrained Action Weights

Fast-VLA's action decoder and head are trained from scratch on task demonstrations, without pretraining on external large-scale action data. Despite this, it achieves competitive results against π0.5 and DM0.5, which both use pretrained action-generating weights (marked "✓" under "Action PT" in Tables 2 and 3). On RoboTwin, Fast-VLA's 92.43% mean is within 1 point of DM0.5's 93.45%. The gap is larger on LIBERO-Plus (79.11% vs 85.90%), suggesting that action pretraining helps most under distribution shift. Teams without access to large-scale action pretraining data can still build competitive policies by leveraging frozen VLM task memory, though they should expect a robustness penalty under perturbation.

6. Overlooked Insights

Online Resident Memory Is Only 1.22 GiB After VLM Offloading

The paper mentions in the Table 2 notes that "Online resident memory after VLM offloading is 1.22 GiB; peak memory during prefill is 12.11 GiB." This means the VLM can be unloaded from memory after the one-time prefill, and the online actor runs with a footprint comparable to a small model. Of this 1.22 GiB, the reusable task memory itself occupies only 174.6 MiB (Appendix C). This is a critical deployment detail: the hardware requirements for the control loop are dramatically lower than the peak, meaning a system could use a high-memory instance for initialization then scale down, or run the actor on a separate lower-cost accelerator.

The Architecture Cannot Handle Mid-Episode Goal Changes

Section E (Limitations) acknowledges: "handling changes in task goals or newly introduced instructions during an episode remains unexplored." Because task memory is fixed at episode start, any instruction change, new object introduction, or task reassignment mid-episode would require a new VLM prefill. The authors suggest "event-triggered memory updates and high-level replanning" as future work. For deployment scenarios where tasks may change dynamically (e.g., human-robot collaboration where a person gives new instructions mid-task), this is a known limitation that would require architectural extension. Investors evaluating Fast-VLA-style approaches should assess whether their target use cases involve static or dynamic task specification.