R3: Training Robots… | arXiv Physical AI Research Summary

1. Key Themes

Free-Form Language Reasoning as Test-Time Compute for Robot Control

The paper's core contribution is demonstrating that a VLM can be trained to produce free-form natural language reasoning before issuing high-level instructions to a frozen low-level robot policy — and that this reasoning genuinely functions as test-time compute, not just a training-time auxiliary signal. The authors show that increasing the reasoning budget at inference time (allowing more tokens) directly improves task success: "allowing a larger reasoning budget generally yields marked gains, especially on group, line, V, iip, gris, iV, diag_line, rect, and iL" (Section 4.2, Table 3). The model also self-regulates reasoning length based on task difficulty: "Figure 5 shows that our model generally elicits longer reasoning traces on harder, lower-success tasks" (Section 4.2).

Two-Stage Post-Training Recipe: Mid-Training + Single-Step RL from Offline Data

R3 uses a practical two-stage pipeline inspired by LLM post-training. Stage I mid-trains a VLM on a small set of expert-generated reasoning traces to initialize the desired reasoning style. Stage II applies single-step rubric-based RL on broader offline data that contains only expert instructions (no reasoning traces), using a VLM-as-judge to reward semantic match between predicted and expert instructions. This avoids expensive multi-turn robot rollouts: "We therefore use a single-step formulation for RL on an expert dataset consisting of an interaction history x_t and the corresponding expert instruction u*_t, with no expert reasoning" (Section 3.3). The recipe is designed to mirror practical data constraints: "high-quality reasoning traces are expensive, while subtask-level instruction labels are easier to obtain" (Section 3.1).

Strong Out-of-Distribution Generalization Over Instruction-Only Imitation Learning

R3 significantly outperforms instruction-only imitation learning (IL) on held-out tasks. On Language Table, R3 beats IL on all five OOD tasks (Section 4.1, Table 2). On the bimanual grocery packing benchmark, R3 (RL only) achieves 47.9% mean success rate vs. 38.0% for IL, and 73.1% normalized progress vs. 65.4% (Section 5, Table 5). The authors attribute this to IL memorizing in-distribution strategies while R3 generalizes learned reasoning behaviors: "IL generalizes badly since it primarily memorizes strategies for in-distribution training data, whereas R3 effectively generalizes the learned reasoning behavior to OOD tasks" (Section 4.1).

Reasoning Causally Contributes to Performance — Not Just Better Representations

The paper provides three pieces of evidence that test-time reasoning itself (not just improved representations from training-time supervision) drives performance gains. First, VQA diagnostics show perception improvements alone don't explain manipulation gains: "Even our best model remains far below Gemini on several VQA categories, yet matches or approaches Gemini on many manipulation tasks" (Section 4.2, Evidence A). Second, adding reasoning as training-time-only supervision (pre-training or co-training) doesn't match R3's performance (Section 4.2, Table 3). Third, truncating reasoning at test time degrades performance proportionally: "Since these variants differ only in the inference-time reasoning budget, this comparison isolates the effect of inference-time reasoning while holding the learned representations fixed" (Section 4.2, Evidence C).

2. Contrarian Perspectives

Structured CoT Templates (Bounding Boxes, Coordinates) Don't Help — Free-Form Reasoning Is Better

Most robotics reasoning approaches (ECoT, SteerVLA, MolmoAct) use structured intermediate representations like object detections, depth-aware perception tokens, or image-space trajectories. R3 shows that adding ECoT-style structured annotations (end-effector state, object states) to reasoning traces actually slightly degrades performance: "Comparing the ECoT variants with their non-ECoT counterparts, we find that adding ECoT components slightly degrades overall performance" (Section 4.4, Table 4). The authors argue this is because their tasks require "reasoning about long-horizon task progress, execution failures, and closed-loop replanning, whereas the additional ECoT components primarily make low-level visual grounding information explicit" (Section 4.4). This challenges the prevailing assumption that more structured, grounded intermediate representations are always better for robot reasoning.

Test-Time Reasoning Provides Real Benefits — Contradicting Prior Robotics Findings

Prior work in robotic reasoning (specifically Chen et al. [8]) concluded that reasoning at test time is not essential and that its primary benefit comes from training-time supervision. R3 directly contradicts this: "In fact, Chen et al. [8] show that using reasoning at test time is not essential, and that its primary benefit comes from providing additional training-time supervision. These findings somewhat contrast with those in LLMs" (Section 1). The paper provides controlled evidence that removing reasoning at test time from the same trained checkpoint significantly degrades performance, and that scaling reasoning tokens up improves success — findings more aligned with LLM test-time compute scaling than with prior robotics work.

RL from Offline Data with Surrogate Rewards Can Work Without Robot Rollouts

The conventional approach to improving robot policies via RL requires online environment interaction with task-success rewards. R3 sidesteps this entirely by using single-step offline RL where the reward is semantic match between the model's predicted instruction and the expert's instruction, judged by a VLM. This is a surrogate for task success: "While this reduces the need for multi-turn robot rollouts, it also optimizes a surrogate objective" (Section 6, Limitations). Yet it works — RL alone (without mid-training) already "substantially improve[s] performance on training tasks" (Section 4.1, Result 1). This suggests that for high-level reasoning, cheaper surrogate rewards can substitute for expensive online rollouts, at least for initial capability gains.

3. Companies Identified

Physical Intelligence (π0.5, π0.7)

Description: Robotics foundation model company developing vision-language-action models for generalist robot control.
Why relevant: π0.5 is used as the low-level policy for the bimanual grocery packing experiments. π0.7 is referenced as a steerable generalist model that can be guided with subtask instructions and visual subgoals. R3's hierarchical architecture (high-level reasoner + frozen low-level policy) is directly applicable to Physical Intelligence's steerable models.
Quote: "We use the dataset of human teleoperation data labeled with instructions directly from Anonymous [2], and fine-tune π0.5 [27] on this data to obtain a steerable low-level policy" (Section 5).

Google DeepMind (Gemini Robotics, Gemini 3 Flash)

Description: AI research lab developing foundation models for robotics and multimodal reasoning.
Why relevant: Gemini 3 Flash is used as the "human expert" to generate reasoning traces and collect trajectories for Language Table. Gemini Robotics is referenced as a generalist robot policy. The paper shows that a 4B model trained with R3 can approach Gemini's performance on manipulation tasks despite being far behind on static VQA — suggesting R3's training recipe can close the gap between small deployable models and large frontier models.
Quote: "We use Gemini 3 Flash as the 'human' expert and construct two data subsets based on the supervision exposed to the learner" (Section 3.1).

Alibaba (Qwen)

Description: AI company producing open-source language and vision-language models.
Why relevant: Qwen3.5-4B is the base model for all R3 experiments, and Qwen3.5-35B-A3B serves as the VLM judge for RL rewards. The choice of a 4B model is explicitly motivated by real-time robotic control constraints: "even the strongest open-source VLMs at model sizes suitable for real-time robotic control, i.e., under 10B parameters, did not naturally produce the style of reasoning needed" (Section 3.2).
Quote: "We use Qwen3.5-4B as the base model for training" (Section 4).

UFACTORY (xArm-7)

Description: Robotic arm manufacturer.
Why relevant: The bimanual grocery packing environment uses dual xArm-7 arms with modified gripper fingers. This is a commercially available platform, making the experimental setup relevant to real-world deployment scenarios.
Quote: "The grocery-packing task suite [2] is built in the MuJoCo simulator [55] using two 7-DoF UFACTORY xArm-7 arms with modified gripper fingers [37]" (Appendix A.2).

NVIDIA (GR00T)

Description: Foundation model for generalist humanoid robots.
Why relevant: Referenced as a generalist robot policy that uses VLM backbones but does not incorporate explicit reasoning. R3's approach could potentially be applied as a post-training layer on top of such models.
Quote: "Recent generalist robot policies and vision-language-action models, including RT-2, Octo, OpenVLA, GR00T, GR-3, and Gemini Robotics, use VLM backbones pretrained on internet data but do not incorporate explicit reasoning" (Section 2).

4. People Identified

Lehong Wu

Lab/Institution: Carnegie Mellon University (AIRe and RCHI labs)
Why notable: Lead author of the paper. The work is supported by ONR, Schmidt Sciences AI2050 Early Career Fellowship, and a TRI U3.0 project, indicating recognition from both government and industry funders.
Quote: Corresponding author, lehongw2@andrew.cmu.edu.

Aviral Kumar

Lab/Institution: Carnegie Mellon University
Why notable: Senior author and likely PI. Recipient of Schmidt Sciences AI2050 Early Career Fellowship. Has prior work on test-time compute optimization for LLMs (referenced as [45] on meta reinforcement fine-tuning), bringing LLM reasoning expertise into robotics. This cross-domain expertise is central to R3's approach of adapting LLM post-training recipes (mid-training + RL) to robotic reasoning.
Quote: Co-author on multiple referenced works including recursive introspection [44] and test-time compute optimization [45].

Zackory Erickson

Lab/Institution: Carnegie Mellon University
Why notable: Co-author with prior work on the bimanual grocery packing environment (RaC [24]), which is used as the second experimental domain. This suggests an established research program in long-horizon bimanual manipulation.
Quote: Co-author on RaC: "Robot learning for long-horizon tasks by scaling recovery and correction" [24].

Yuxiao Qu

Lab/Institution: Carnegie Mellon University, Amazon AI PhD Fellow
Why notable: Co-author with expertise in LLM reasoning and mid-training (referenced works [44, 45] on recursive introspection and test-time compute optimization). Brings the LLM post-training methodology that R3 adapts to robotics.
Quote: "YQ gratefully acknowledges support from the Amazon AI PhD Fellowship" (Acknowledgments).

Chelsea Finn

Lab/Institution: Physical Intelligence / Stanford University
Why notable: Referenced as co-author on multiple foundational works including π0.5 [27], π0.7 [28], Hi Robot [50], and ECoT [64]. Her work on hierarchical VLA models and steerable policies directly underpins R3's architecture. The comparison with ECoT in Section 4.4 is essentially a comparison with her group's approach.
Quote: Referenced across [27, 28, 50, 64].

Sergey Levine

Lab/Institution: Physical Intelligence / UC Berkeley
Why notable: Referenced as co-author on π0.5, π0.7, RT-2, SayCan, and ECoT. His work on generalist robot policies and language-conditioned manipulation forms the foundation that R3 builds upon and extends.
Quote: Referenced across [1, 5, 8, 27, 28, 64].

5. Operating Insights

You Can Train Robotic Reasoners Without Robot Rollouts — Use Offline Data + VLM Judges

For teams building hierarchical robot systems (high-level planner + low-level controller), R3 offers a practical training pipeline that avoids the cost and complexity of online RL. The key insight is that you can train a high-level reasoner using only offline demonstration data with instruction labels, using a VLM-as-judge to provide semantic rewards for single-step RL. The grocery packing experiment shows this works even without mid-training on reasoning traces: "we skip Stage I mid-training because the base VLM already produces useful reasoning on this domain" (Section 5). For teams with existing teleoperation datasets labeled with subtask instructions, this means you can potentially add reasoning capabilities without collecting any new data or running robot rollouts. The reward function is simple to implement: exact string match for structured instruction sets, or VLM-as-judge for open-ended instruction spaces (Appendix A.4).

Mid-Training on Reasoning Traces Is Cheap but High-Leverage — Even 1/4 of the Data Recovers Most Benefits

The paper shows that mid-training on a small set of expert reasoning traces provides a strong behavioral prior that makes subsequent RL more reliable. Critically, even 1/4 of the mid-training data recovers most of the benefit: "although R3 (1/4th mid) slightly underperforms full R3 on T_M, it already matches or exceeds full R3 on T_R and T_O" (Section 4.1, Result 2). For a company, this means you don't need to collect massive reasoning-annotated datasets — a modest investment in reasoning trace generation (even using a frontier VLM like Gemini as the "expert") can seed the reasoning style, and broader instruction-only data can do the rest via RL. The authors even note that a protocol recording human teleoperator "stream of consciousness" and rewriting it with VLMs could work (footnote 1, Section 3.1).

Choose Your Base VLM Size for Real-Time Control, Then Train Reasoning Into It

The paper explicitly chose a 4B parameter model because of real-time control constraints: "model sizes suitable for real-time robotic control, i.e., under 10B parameters" (Section 3.2). Off-the-shelf VLMs at this scale don't naturally produce useful robotic reasoning — their reasoning is "often shallow" and exhibits "thought-switching, a failure mode related to 'underthinking'" (Section 3.2). The implication for deployment teams is that you cannot expect a small VLM to reason well about manipulation out of the box, but R3's post-training recipe can transform it into an effective reasoner. The gap between the 4B R3 model and Gemini on static VQA tasks, yet near-parity on manipulation tasks, suggests this training recipe extracts disproportionate operational value from small models.

6. Overlooked Insights

History Context Is Essential for Long-Horizon Reasoning — and Simple Previous-Response Format Works

The paper finds that providing interaction history (the full previous response from the high-level VLM) consistently improves performance: "from 44.9% to 51.0% on line, and from 52.3% to 57.6% on V" (Section 3.1, Table 1). This is a simple, implementation-cheap design choice — just include the previous reasoning + instruction as context — but it has outsized impact because it allows the model to "carry forward its inferred progress and plan" (Section 3.1). For teams building hierarchical controllers, this means the high-level reasoner needs memory of its own previous outputs, not just the current observation. On grocery packing, the authors found that using just the previous instruction (without reasoning) as history works comparably, simplifying implementation further (Section 3.3, RL design choices for grocery packing).

RL Tends to Collapse Rare but Useful Behaviors — Data Filtering for Repetitive Steps Is Critical

A practical finding buried in the RL design choices: the VLM reasoner learned to repeat the previous instruction as a reward shortcut during RL, because "many expert trajectories contain repeated instructions, and repetition can become a severe reward shortcut during RL" (Section 3.3). The authors solved this by filtering repetitive steps from RL data. Additionally, "rare instructions (Separate, Touch, and ArmMoveAbs) collapse after RL, suggesting that behaviors misaligned with the expert are dropped in RL" (Section 4.3). This has implications for deployment: if your expert data has a long tail of rare but important recovery behaviors, single-step RL with surrogate rewards may eliminate them. The mode-seeking nature of RL (Figure 20) means it concentrates on high-reward strategies, which could reduce behavioral diversity needed for robustness in unstructured environments.