How Should Vision-Language-Action Models Use Proprioceptive State?

1. Key Themes

No Universal Best Way to Feed Robot State into VLA Models

The paper demonstrates that there is no task-agnostic best interface for feeding proprioceptive state (joint angles, end-effector pose) into Vision-Language-Action (VLA) models. Different types of manipulation tasks favor entirely different architectural wiring. For example, "Family A rearrangement tasks favor sp (68.7%, +7.0 over no-state)... Family B articulation tasks reverse this order: vp leads at 68.8% (+6.1)... Family C knob-and-switch control... se leads at 42.8% (+3.3)" (Section 5.2, RQ1). For companies deploying generalist robots, this means a one-size-fits-all state interface will systematically underperform on specific manipulation categories like precision knob-turning or heavy rearrangement.

Short State History Helps, Long History Hurts

Providing a short history of robot state (around 8 frames) improves performance, but feeding long uncompressed histories (up to 96 frames) actively degrades control. "Figure 3 shows a clear non-monotonic trend: short histories improve performance over the corresponding single-frame models, whereas deeper uncompressed histories provide no additional benefit and eventually degrade control" (Section 5.2, RQ2). This implies that simply stacking more temporal data into a transformer context window is counterproductive, hurts precision tasks, and wastes compute.

Optimal State Injection Route Depends on Temporal Budget

Where the state enters the model architecture should change dynamically based on whether you are using a single frame or a short history. "With a single frame (K=1), state favors the VLM side... With a short history (K=8), the advantage shifts decisively to the action side" (Section 5.2, RQ3). The distilled design rule is: "inject single-frame state into the VLM, but route multi-frame state history to the action head" (Section 5.2, RQ3).

2. Contrarian Perspectives

More Temporal Context is Not Always Better

In machine learning, the prevailing assumption is that more context equals better performance. This paper challenges that for robot state, showing that long histories cause a "pronounced drop" in performance, especially for precision tasks (Section 5.2, RQ2). Companies building memory-heavy robot policies or relying on massive observation histories should be wary of blindly feeding long state sequences into their models, as it introduces noise that interferes with action prediction.

Discrete Text Tokenization of State is Computationally Prohibitive

Some leading models, like Physical Intelligence's π0.5, serialize robot state into text tokens. While this yields good macro performance, the paper reveals it is the most computationally expensive approach by a massive margin. "sp serializes the state into roughly 66 discrete prompt tokens, adding about 1114 training GFLOPs per sample... the most expensive design by two orders of magnitude on the training side" (Section 5.2, RQ1). Meanwhile, a dedicated state expert (se) achieves nearly the exact same macro success rate (57.6% vs 57.7%) at a tiny fraction of the compute (2.6 GFLOPs/sample). Startups should avoid text-serializing state if they care about training and inference efficiency.

3. Companies Identified

Physical Intelligence

Description: Creator of the π0 and π0.5 Vision-Language-Action models. Why relevant: The paper uses π0.5 as its base testbed architecture. The paper also critiques π0.5's method of serializing state into text tokens as computationally inefficient. "We adopt π0.5 (Intelligence et al. 2025)—a VLM coupled with a flow-matching action expert... as the base policy for the entire study" (Section 4.1). "π0.5 serializes it into discrete text tokens in the VLM prompt" (Section 3.2).

NVIDIA

Description: Creator of the GR00T N1 foundation model for humanoid robots. Why relevant: GR00T N1's approach to state injection is cited as an alternative architecture where state bypasses the VLM. "GR00T N1 feeds a state embedding directly to the action head" (Section 3.2).

OpenVLA / OpenVLA-OFT

Description: Open-source VLA model and its optimized fine-tuning variant. Why relevant: OpenVLA-OFT represents a third distinct approach to state handling, projecting it continuously into the language model sequence. "OpenVLA-OFT projects it continuously into the language-model sequence" (Section 3.2).

4. People Identified

Yiren Zhao

Lab/Institution: The Hong Kong University of Science and Technology (Guangzhou) / AI2 Robotics X-Lab. Why notable: Lead author of the study, driving the systematic evaluation of VLA state interfaces.

Weiyu Guo

Lab/Institution: Multimedia Laboratory (MMLab), The Chinese University of Hong Kong / AI2 Robotics X-Lab. Why notable: Corresponding author, indicating a leading role in directing the research.

Yandong Guo and Rushi Dai

Lab/Institution: AI2 Robotics X-Lab / HKUST(GZ). Why notable: Corresponding authors representing the intersection of academic robotics and applied AI labs.

5. Operating Insights

Adopt an 8-Frame State History as a Practical Default

For engineering teams building VLA models, the paper provides a concrete, empirically backed starting point for temporal context. "An eight-frame history provides a practical operating point... compact histories are consistently more reliable than long raw sequences" (Section 5.2, RQ2). On complex, multi-stage composite tasks, moving from a 1-frame to an 8-frame history via the action prefix improved success rates by 10.8 percentage points (Table 2).

Route State Based on Frame Count to Maximize Efficiency

CTOs should implement a dual-path strategy for state injection rather than a single hardcoded route. If using only the current frame, inject it into the Vision-Language Model (VLM) prefix so it can contextualize the visual scene. If using a short history (e.g., 8 frames), route it directly to the action expert. "inject single-frame state into the VLM, but route multi-frame state history to the action head" (Section 5.2, RQ3). This prevents the VLM from being overloaded with temporal tokens and allows the action head to directly leverage state evolution, yielding the highest performance gains.

6. Overlooked Insights

State History Gains Come from Temporal Variation, Not Added Model Capacity

A common pitfall in behavioral cloning is that models use extra input tokens as a crutch (copying their own recent trajectory) rather than learning true temporal dynamics. The paper proves the gains are real by using a "repeat-current control" where the current state is simply copied into 8 slots. "The repeated-state control falls substantially short of the genuinely ordered history... the improvement cannot be explained by additional conditioning slots alone; it arises from the temporal variation encoded across successive states" (Section 5.2, RQ2). This means teams can confidently invest in temporal state encoding knowing it provides genuine signal, not just a shortcut for the model.

State History is Most Critical During Subtask Transitions

The case study on the "PrepareToast" task reveals that the 8-frame history policy's advantage over the 1-frame policy appears specifically when the robot must transition from placing items to closing a cabinet. "The separation begins after both items have been placed... ap8 reaches the cabinet-reclosed milestone S3 in 56% [vs 30% for ap1]" (Appendix B.2). Furthermore, the policy is "most sensitive to ordered state variation around progress boundaries" (Appendix B.4). This suggests state history is primarily useful for helping the robot figure out when one phase of a task is done and the next must begin, rather than for fine-grained continuous control.