Recovering Aggressively Pruned Vision-Language-Action Models with Offline Hidden-State Distillation

1. Key Themes

Fully Offline Recovery of Pruned VLA Models

The paper demonstrates that aggressively pruned Vision-Language-Action (VLA) models can be recovered almost entirely offline, eliminating the need for expensive reinforcement learning (RL) or simulator-in-the-loop training. By pruning the width of the model (attention heads and MLP channels) while keeping the residual stream the same size, the student model's hidden states can be directly matched to the teacher's. The authors state: "We recover most of the lost success entirely offline... Training against a cache built in one teacher pass lifts the 63%-reduced student to within 3.5 points of the teacher in about 8 GPU-hours" (Abstract). This contrasts with prior RL-based methods requiring "about 320 GPU-hours of training" (Section I).

Width Pruning Preserves Task Success Over Depth Pruning

The research provides a controlled comparison between width pruning (narrowing blocks) and depth pruning (removing layers). The findings reveal a clear trade-off: width pruning yields higher task success, while depth pruning yields lower latency. The paper notes: "Width pruning achieves higher success than the CKA-guided depth baseline at every measured point on both backbones... Depth and width pruning differ qualitatively before recovery. Depth-pruned models reach 0% success at all three OpenVLA-OFT points, while width pruning at 45% retains 54.4%, so recovery begins from a functioning policy" (Section V-D).

Real-World Deployment Viability and Speedups

The paper validates its approach on a physical 6-DoF manipulator, proving that compressed models can run faster and more efficiently on robot hardware without sacrificing capability. The distilled student at 72% reduction "reaches 77.5% success against 59.5% for supervised recovery, runs 2.23x faster on-board than the teacher, and uses 62% less memory" (Abstract). On the robot's on-board Jetson Thor, latency dropped from 362 ms to 162 ms (Table III).

Identifying the Compression Sweet Spot

The authors conducted a sweep over nine compression ratios to determine exactly when recovery methods begin to matter. They found that up to 45% reduction, simple supervised fine-tuning is sufficient. However, beyond 63% reduction, hidden-state distillation becomes critical. The paper states: "Up to 45% reduction the two do not differ significantly on OpenVLA-OFT. Hidden-state distillation then adds +2.1 to +4.5 points there between 63% and 87%, and +9.4 to +22.1 points on CogACT from 63% onward" (Abstract).

2. Contrarian Perspectives

Reinforcement Learning is Unnecessary for Recovering Pruned Policies

Conventional wisdom in robotics, as established by prior work like RLRC, suggests that recovering a heavily pruned VLA requires a two-stage process: supervised fine-tuning followed by reinforcement learning to close the residual gap. This paper challenges that notion, arguing that the gap can be closed entirely offline. The authors note: "RLRC [7] recovers a 90%-pruned model... in two stages... That second stage needs a simulator or a robot in the loop, a reward to optimize, and about 320 GPU-hours of training... We eliminate most of this gap offline" (Section I). They prove that "A pruned vision-language-action model can be recovered from a cache of demonstrations and teacher hidden states alone, without reinforcement learning, rollouts, or a reward" (Section VIII).

Compressed Models Can Outperform the Dense Teacher in the Real World

Most practitioners assume that model compression inherently degrades performance, expecting a compressed student to perform worse than the dense teacher. This paper presents evidence to the contrary in real-world deployment. On the physical robot, the 72%-reduced distilled student actually outperformed the teacher. The authors report: "Over 200 paired episodes the SFT+KD student reaches 77.5%, outperforming the SFT student by 18.0 points (p < 0.001). It also exceeds the teacher’s 65.5% by 12.0 points (p = 0.004)" (Section V-F). This suggests that distillation may act as a form of regularization that improves robustness.

3. Companies Identified

OpenVLA / OpenVLA-OFT

Description: An open-source vision-language-action model pairing a Llama-2 7B backbone with a continuous L1 head. Why relevant: It serves as one of the two primary base VLA models used for pruning and recovery experiments in the paper. Quotes: "OpenVLA-OFT [2] pairs a Llama-2 7B backbone with a continuous L1 head emitting 8-step action chunks" (Section II).

CogACT

Description: A foundational VLA model that uses a Prismatic VLM family backbone with a diffusion transformer action head. Why relevant: It is the second primary base VLA model used in the experiments, demonstrating that the recovery method generalizes across different action-head paradigms. Quotes: "CogACT [3] keeps the same VLM family and replaces the head with a diffusion transformer" (Section II).

AgileX

Description: A robotics company that manufactures manipulator arms. Why relevant: Their PiPER 6-DoF arm was used for the real-robot validation of the compressed models. Quotes: "On an AgileX PiPER 6-DoF arm (Fig. 5) we evaluate the teacher and the 72% OpenVLA-OFT students" (Section V-F).

NVIDIA

Description: Technology company known for GPUs and edge computing platforms. Why relevant: The robot's on-board compute was an NVIDIA Jetson Thor, used to measure real-world latency and memory savings. Quotes: "Latency and memory are measured on the robot’s on-board Jetson Thor (Table III)" (Section V-F).

4. People Identified

Chiyoung Kim, Sanghyuk Roy Choi, and Minhyeok Lee

Lab/Institution: Chung-Ang University, Seoul, Republic of Korea. Why notable: The authors of the paper who developed the offline hidden-state distillation method for VLA recovery. Quotes: "The authors are with Chung-Ang University, Seoul, Republic of Korea" (Section I).

Y. Chen, Y. Han, Y. Huang, and X. Li (RLRC authors)

Lab/Institution: Not explicitly stated in the provided text, but referenced as prior work [7]. Why notable: They developed the RL-based recovery method (RLRC) that this paper directly challenges and improves upon. Quotes: "RLRC [7] recovers a 90%-pruned model of this backbone to its dense LIBERO score in two stages" (Section I).

5. Operating Insights

Cache the Teacher's Hidden States for Cheap Recovery

CTOs and heads of engineering should adopt the teacher caching strategy for model compression. Instead of running the large teacher model inside the training loop (which costs memory and recomputes targets), run it once to cache the inputs, hidden states, and action targets. The paper states: "Running the teacher inside the training loop costs memory and recomputes the same targets at every epoch, so we run the teacher once. A single offline pass over the recovery data stores the teacher’s hidden states... The student is then trained against this cache alone" (Section III-C). This allows recovery to run on a single 48 GiB GPU in about 8 hours.

Match Hidden States, Not Just Actions, for Deep Compression

When compressing a VLA beyond 45%, standard supervised fine-tuning on action outputs is insufficient because the action output is too low-dimensional to constrain the internal representation. Engineers should use hidden-state distillation (matching the internal representations directly) to prevent representation drift. The authors explain: "The action loss reaches the representation only through the action output, so the more pruning disturbs that representation, the less the supervised term constrains it" (Section V-B). They found that hidden-state distillation reduces the student's mean-squared error to the teacher's cognition feature from 0.61 to 0.14 on held-out data (Fig. 3(c)).

Choose Pruning Strategy Based on Hardware Constraints

Engineers must make a deliberate choice between width and depth pruning based on their specific deployment hardware. If task success is the priority, width pruning is superior. If latency is the absolute bottleneck, depth pruning is better. The paper notes: "Width pruning and depth pruning thus present a trade-off, one preserving task success and the other yielding larger latency reductions" (Section V-D). However, they also found that width pruning yields more speedup on robot hardware than on workstations: "On the Jetson Thor, width pruning yields about twice the speedup measured on the workstation while keeping its success advantage" (Section VI).

6. Overlooked Insights

Latency Does Not Strictly Follow Parameter Count

A buried finding is that reducing parameters does not always reduce latency proportionally, especially for models with diffusion-based action heads. For CogACT, latency hits a floor because the diffusion sampler runs a fixed number of steps regardless of backbone size. The paper states: "Latency does not follow the parameter count (Table II). On CogACT latency falls from 129.1 ms to 85.0 ms at 72% reduction and then stays at 84.7–85.0 ms. The diffusion sampler runs ten DDIM steps independently of backbone size, and that fixed cost sets a latency floor that further narrowing cannot lower" (Section V-E). This means engineers must identify the latency floor before over-compressing and unnecessarily sacrificing model capacity.

Distillation Advantage Persists Under Distribution Shift

An important but easily overlooked finding is that the robustness gained through hidden-state distillation carries over to unseen perturbations. The compressed student does not just memorize the recovery distribution; it generalizes better than supervised recovery. The authors report: "The advantage of the KD term persists under perturbation. Over the 1,992 SimplerEnv variant episodes on CogACT, the 72% SFT+KD student reaches 60.7% against 37.2% for the SFT student and 66.6% for the teacher. The margin of 23.5 points is as large as the +22.1 points measured in distribution" (Section V-F). This indicates that hidden-state distillation inherently improves the robustness of the policy.