Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models

1. Key Themes

RL Post-Training Unlocks Significant Performance Gains Over Imitation Learning

The paper demonstrates that moving beyond supervised fine-tuning (SFT) and behavior cloning to reinforcement learning (RL) provides a massive leap in task success. By applying their GRPO post-training framework, the authors improved the average success rate on 24 standard RoboCasa manipulation tasks from 67.4% to 80.6%. As stated in Section 4.2, "Z-1 RL improves the average success rate from 67.4% to 80.6%, corresponding to a gain of 13.2 percentage points." This proves that allowing a robot to learn from its own simulated failures is a highly effective way to refine manipulation skills after initial training on human demonstrations.

Efficient Rollout Construction via Shared Prefixes and Tree Branching

Generating rollouts (robot practice attempts) in simulation is computationally expensive, especially when comparing multiple variations of a full trajectory. Z-1 tackles this by sharing the "approach" phase of a trajectory across multiple rollouts and only branching out during the critical "manipulation" phase (e.g., the actual grasping or turning motion). The authors note in Section 2.2 that "Z-1 constructs rollout groups with shared prefixes and tree-structured branching to reduce redundant prefix execution and focus comparisons on task-critical segments." This drastically cuts down on wasted simulation compute and provides a cleaner learning signal for the parts of the task that actually matter.

Selective Joint Training of Vision and Action Modules

Most VLA-RL pipelines freeze the vision-language model (VLM) during RL to maintain stability, only updating the action generation module. Z-1 challenges this by selectively unfreezing the VLM for tasks where the robot is failing due to poor visual understanding rather than poor motor control. The authors explain in Section 4.3.2 that "Selective joint training allows the model to adapt perception, grounding, and action generation together, leading to more stable and effective improvement." This allows the system to fix its own visual recognition errors on the fly.

Success-Aware Reward Calibration for Better Credit Assignment

In robotics RL, tasks usually have binary rewards: success or failure. However, treating all successful attempts the same provides a weak learning signal. Z-1 introduces a "Success-Aware Reward Decay" that gives higher rewards to robots that complete the task faster. According to Section 3.4, "we use Success-Aware Reward Decay to introduce an ordering among successful rollouts according to completion time." This encourages the robot to not just succeed, but to find more efficient paths to success without requiring complex, hand-engineered dense reward functions.

2. Contrarian Perspectives

You Do Not Need Proprietary Data to Achieve State-of-the-Art Performance

Many robotics companies believe that achieving top-tier performance requires massive, privately collected, teleoperated datasets. Z-1 achieves state-of-the-art results on the RoboCasa benchmark using only publicly available data and a publicly available base model ($\pi_{0.5}$). The authors state in the Abstract: "These results show that systematic GRPO post-training can substantially improve flow-based VLA policies without additional private demonstrations." This suggests that algorithmic improvements in RL post-training can substitute for raw data scale, lowering the barrier to entry for new robotics companies.

Freezing the Vision-Language Backbone During RL is Suboptimal

The conventional wisdom in VLA-RL is to freeze the large vision-language model during RL fine-tuning to prevent catastrophic forgetting and maintain training stability. Z-1 argues this leaves performance on the table for complex tasks. As noted in Section 3.4, "AE-only adaptation can be insufficient for tasks with low SFT success or perception-sensitive failures, where the limiting factor may be inaccurate object localization, weak spatial grounding, or poor alignment between vision-language representations and action generation." By selectively unfreezing the VLM, Z-1 proves that joint perception-action optimization is necessary for tasks requiring precise visual grounding.

3. Companies Identified

Physical Intelligence (PI)

• Description: Creators of the $\pi_0$, $\pi_{0.5}$, $\pi_{0.6}^*$, and $\pi_{0.7}$ Vision-Language-Action models.
• Why relevant: Z-1 is built directly on top of Physical Intelligence's $\pi_{0.5}$ base model. The paper validates and extends PI's architecture by showing how it can be efficiently improved via RL. "Built on top of $\pi_{0.5}$, Z-1 uses only publicly released RoboCasa demonstrations for SFT..." (Abstract).

NVIDIA

• Description: Creators of the GR00T and GR00T N1.5 generalist robot foundation models, as well as the RoboCasa simulation benchmark.
• Why relevant: NVIDIA's models serve as the primary baselines for comparison. Z-1 outperforms them significantly. "The GR00T and GR00T N1.5 results are taken from their reported RoboCasa evaluations (NVIDIA et al., 2025)" (Section 4.1.4). Z-1 beat GR00T by 30.9 percentage points and GR00T N1.5 by 20.9 percentage points (Section 4.2).

Zioneer Robot Team

• Description: The research team (authors) behind the Z-1 paper.
• Why relevant: They are the entity publishing this SOTA RL post-training framework. "Zioneer Robot Team" (Title page).

4. People Identified

Lang Cao & Renhong Chen

• Lab/Institution: Zioneer Robot Team
• Why notable: They are the core drivers of the research, responsible for the conceptualization and implementation of the RL framework. "Lang Cao and Renhong Chen equally contributed to this paper and are responsible for the proposal and implementation of the RL idea." (Appendix A).

Yitong Li

• Lab/Institution: Zioneer Robot Team
• Why notable: Project supervisor who guided the research direction and technical development. "Yitong Li supervised the project and provided guidance on research direction and technical development." (Appendix A).

5. Operating Insights

Invest in RL Post-Training Pipelines Rather Than Just More Data Collection

For CTOs and heads of engineering, this paper makes a clear case that the highest ROI for improving robot performance right now is building a robust RL post-training pipeline, not just collecting more teleoperation data. Z-1 achieved a 13.2 percentage point gain purely through simulated RL without a single new human demonstration. As stated in Section 4.2, "These results show that systematic GRPO post-training can substantially improve flow-based VLA policies without additional private demonstrations." Building infrastructure for simulated rollouts and GRPO optimization should be a top engineering priority.

Adopt Task-Specific RL Configurations Instead of One-Size-Fits-All

Z-1 does not use a single RL recipe for all tasks. Instead, it dynamically selects which modules to enable (e.g., shared prefixes, VLM joint training) based on early training diagnostics. The authors note in Section 3 that "Each task uses a fixed subset of Z-1 modules, chosen before final evaluation based on training-stage diagnostics... The available modules include Shared-Prefix GRPO, Tree-Structured Prefix Branching, Success-Aware Reward Decay, and VLM–Action Expert joint training." Operators should build diagnostic tools to identify failure modes (e.g., perception failure vs. motor control failure) and route tasks to the appropriate RL configuration.

6. Overlooked Insights

Flow-SDE Formulation is the Key to RL for Flow-Based Policies

Applying standard policy gradient methods to flow-matching models (like $\pi_0$) is mathematically difficult because the actions are generated via a deterministic integration path, not sampled from a standard probability distribution. Z-1 solves this by injecting Gaussian noise into the flow transitions, turning it into a stochastic process (Flow-SDE). As detailed in Appendix D.3, "A direct application of policy-gradient methods to deterministic flow sampling is non-trivial... To obtain a tractable policy ratio, we follow the flow-SDE formulation... This converts the deterministic flow generation process into a stochastic Markov process by injecting Gaussian noise..." This technical detail is crucial for any engineering team looking to apply RL to modern flow-based VLA architectures.

Penalizing Slow Success Can Hinder Overall Learning

When calibrating rewards based on completion time, it is tempting to actively penalize slow successful rollouts to encourage efficiency. However, Z-1 found that this can backfire. The authors compared their Success-Aware Reward Decay against a "Reward Penalty" baseline and found that "Reward Penalty substantially reduces average completion steps... At the same time, its success rate increases more slowly, suggesting that penalizing successful but slower rollouts can make optimization overly aggressive when the policy is still learning reliable task completion" (Section 4.3.3). For operators, this means reward shaping should prioritize achieving reliable success first, using non-negative ordering for efficiency rather than explicit penalties.