Teahose.
SIGN IN
NEW HERE — WHAT TEAHOSE DOES
We read the entire AI & tech firehose — so you don't have to.
PODPodcastsAll-In, No Priors, Acquired…
NEWNewslettersStratechery, Newcomer…
PAPPapersPhysical AI research
PHProduct Huntdaily launches
VCInvestor ScoutSequoia, a16z, Benchmark…
CLAUDE DISTILLS →
7 reads, 30 sec each — free, 6 AM ET.
+ a live graph of the companies, people & themes underneath.
HOME/ARXIV PHYSICAL AI RESEARCH/Training Reasoning Models in Sim…
PAPR
// RESEARCH PAPER
ARXIV PHYSICAL AI RESEARCH

Training Reasoning Models in Simulation to Natively Use Real Robots

DATE November 13, 2026SOURCE ARXIV PHYSICAL AI RESEARCHPARTICIPANTS JIAMING TANG, SONG HAN, ET AL. (ARXIV PHYSICAL AI)ARXIV 2611.18562
// KEY TAKEAWAYS5 ITEMS
  1. 01A frontier-style "robot-use agent" can be trained with RL alone in simulation, no demonstrations
  2. 02Sim-to-real transfer works zero-shot on a robot the model never saw
  3. 03Reasoning-based policies generalize to new goals where demonstration-trained VLAs collapse
  4. 04Robot skills can be added without degrading general intelligence
  5. 05Curriculum plus transfer solves the zero-reward cold-start problem
// SUMMARY

1. Key Themes

A frontier-style "robot-use agent" can be trained with RL alone in simulation, no demonstrations

The core contribution is showing that an open-weight 27B reasoning model (Qwen3.8-27B) can be turned into a robot controller using only reinforcement learning with a binary success reward. Abstract: "an open-weight 27B reasoning model can be trained to use real robots natively through reinforcement learning, entirely in simulation and without demonstrations." The model sees camera frames and arm state, thinks in text, and emits the next end-effector pose, with inverse kinematics and the robot's own controller doing the rest. On LIBERO, "training raises the success rate from 1.6% to 74.4%" (Abstract; Section 4.1), close to GPT-6.1 Sol (76.2%) and GPT-6 Astra (80.6%) at max effort. For operators: the training signal is a pass/fail from a simulator, which is far cheaper than teleoperated data collection.

Sim-to-real transfer works zero-shot on a robot the model never saw

The headline real-world result: on a dual-arm YAM, "the final model succeeds in 18 of 30 trials, whereas the base model succeeds in none" (Section 4.4; Table 7). The LIBERO-only checkpoint, trained solely on single-arm Franka sim tasks, still hit 7 of 30. The mechanism is the interface: "Because the model specifies poses rather than joint commands and each robot's own inverse kinematics and controller execute them, the same interface applies in simulation and on a real robot" (Section 1). The robot-specific parts live outside the model, in a prompt describing the frame and cameras plus the robot's own IK/controller (Section 3.1). Embodiment portability comes from the abstraction layer, not from the model weights.

Reasoning-based policies generalize to new goals where demonstration-trained VLAs collapse

This is the sharpest comparison in the paper. On LIBERO-Pro's task perturbation (same scene, new instruction), "π0.5 falls from 96.9% on LIBERO to 0.75%, a drop of 96.2 points, while our model falls from 74.4% to 57.4%, a drop of 17.0 points" (Section 4.2; Figure 4b). The authors' diagnosis: "π0.5 tends to carry out the task it was trained on in that scene, while our model does what the new instruction asks." The caveat, also from Section 4.2: π0.5 still wins on perturbations that preserve the original motion (reworded instructions, restyled objects, Table 3: 95.8% and 96.0% versus 73.3% and 60.2%). So this is a trade-off between in-distribution reliability and goal flexibility, not a blanket win.

Robot skills can be added without degrading general intelligence

"Robot-use training causes no measurable loss in performance on nine reasoning, coding and vision benchmarks" (Section 4.5; Figure 8; Table 6). Mean change across the nine is −0.15 points and every 95% interval includes zero. The one soft spot is ERQA (embodied reasoning), which fell 2.0 points (68.5 to 66.5, CI [−6.2, +2.2]). The strategic implication is that robot-use RL can be a post-training stage in a foundation model pipeline rather than a separate specialist model.

Curriculum plus transfer solves the zero-reward cold-start problem

With a binary reward, tasks with near-zero base success give no gradient. The authors train first on tasks that already yield signal and let transfer unlock the rest: "the base model succeeds in about 2% of episodes on LIBERO's Spatial, Object and Goal suites, in 0.4% on its long-horizon suite LIBERO-Long" (Section 3.3). After training on the three easy suites, zero-shot LIBERO-Long rises to 39.0% (Section 4.3; Figure 5a), and adding it to training pushes it to 53.2%. Skill transfer between task families is what makes sparse-reward robot RL tractable.

2. Contrarian Perspectives

You may not need a harness, perception stack, or skill library

Most open robot-agent work wraps models in scaffolding. The paper lists "ORA, ASPIRE, Harness VLA, OpenRUA, RAX, uni-robot-agent and PyRUA-Lean" (Section 2) and argues harnesses have costs: one "may restrict their action space and thereby make other tasks infeasible. As models become more capable, some components of the harness may also be absorbed into the models themselves" (Section 1). Their model uses "no perception tools, grasp or motion planners, or skill libraries, so the model itself has to localize the objects in the camera frames, determine how to reach and manipulate them, and detect when a move has failed" (Section 3.1). Companies whose moat is a hand-built modular stack should note the direction of travel, though this paper only tests on tabletop manipulation.

Demonstrations may not be the best way to get generality, and robotics data may belong in pretraining, not SFT

The authors deliberately skip supervised fine-tuning: "robotics data, even when available, should be added at the pretraining and mid-training stages rather than used for SFT on a model that has already been post-trained" (Section 5). They also posit that the π0.5 gap "may come from imitating demonstrations versus reasoning about the task" (Section 4.2). That cuts against the dominant strategy of scaling teleoperated demonstration data for VLAs. It is a hypothesis here, not a tested conclusion, since the paper does not run a controlled ablation of SFT versus RL.

Slow, token-heavy "thinking" before every move may be the source of capability, not a bug

Conventional wisdom says robot control needs low latency and high frequency. This paper finds the opposite for generality: "Most of what it can do on LIBERO therefore comes from reasoning at length before each move" (Section 4.6). At low effort (0.86k thinking tokens/round) success falls to 21.9% versus 74.4% at xhigh (3.23k tokens). Rounds take about 4.3s at 747 tokens/s on a GB300 (Section 4.6). This approach is suited to deliberate, reasoning-heavy manipulation, not dynamic or contact-rich control.

3. Companies Identified

OpenAI — Maker of GPT-6 Astra and GPT-6.1 Sol, the frontier proprietary robot-use agents used as baselines. Relevant because the paper shows an open-weight model closing most of the gap. Quote: "approaching GPT-6 Astra and GPT-6.1 Sol at maximum reasoning effort" (Abstract); Table 2 lists Astra at 80.6% and Sol at 76.2% max effort.

Anthropic — Maker of Claude Fable 5.2, cited as a frontier proprietary robot-use model. Relevant as part of the closed-model competitive set. Quote: "strong performance has so far been achieved only by frontier proprietary models such as GPT-6 Astra and Claude Fable 5.2" (Abstract). It is not evaluated in the paper's experiments.

Physical Intelligence — Developer of the π0 and π0.5 VLAs, the main demonstration-trained baseline. Relevant because their approach is stress-tested on instruction changes. Quote: "On LIBERO-Pro, π0.5 succeeds in only 0.75% of episodes when assigned a new task in a familiar scene whereas our model succeeds in 57.4%" (Abstract). The paper also notes π0.5 "outperforms our model on LIBERO" (96.9% vs 74.4%, Section 4.2). The π0.5 numbers are taken from prior reports, not run by the authors (Section 4).

NVIDIA — Provides Isaac Lab (RoboDojo simulation), the RoboLab benchmark/leaderboard, Cosmos3-Nano-Policy, GR00T N1.6, ModelOpt quantization, KDA for megakernel optimization, and GB300/H100 hardware; an author also interned there. Relevant as both ecosystem and competitor. Quote: "the model trained on LIBERO and RoboDojo outperforms the π0.5 entry on the RoboLab leaderboard (37.3% vs. 28.0%)" (Section 1); it is "on par with Cosmos3-Nano-Policy at 36.8%" and far above GR00T N1.6 at 7.2% (Section 4.3). Note: Cosmos3-Nano-Policy leads on complex tasks, 25.3% vs 13.2% (Table 5).

Inco AI — Affiliation of several authors; supplies the DFlash2 drafter and the Splash local inference engine. Relevant for on-device deployment. Quote: "it decodes 52 tokens/s on an M5 Max in Inco's Splash engine" (Section 4.6). The conflict of interest is worth noting.

Alibaba/Qwen Team — Source of the open-weight Qwen3.8-27B base model. Relevant as the foundation that makes the open-weight path viable. Quote: "We train the open-weight reasoning model Qwen3.8-27B" (Section 4).

Google DeepMind (Gemini Robotics Team) — Cited for the ERQA embodied reasoning benchmark used in the capability-retention check (Section 4.5; Table 6).

DROID dataset ecosystem / Stanford-Berkeley-affiliated groups — The RoboLab leaderboard competitors are "all DROID checkpoints" (Figure 5 caption), meaning they were trained on real-robot data, while this model saw none.

i2rt (implied by "YAM") — The real dual-arm robot is called YAM, though the paper never names its manufacturer, so I am not attributing it.

Hardware/infra layer (NVIDIA GB300/H100/H200, Apple M5 Max, RTX 5090) — Relevant for deployment economics. Quote: "it decodes 221 tokens/s on a single RTX 5090 and 747 tokens/s on a single GB300" and "The model can therefore run locally on a single consumer GPU or a laptop" (Section 4.6).

4. People Identified

Song Han — MIT (also NVIDIA, Inco AI affiliation), corresponding author and senior lead. Notable for efficient AI systems and model compression, which explains the paper's emphasis on KV-cache offload, quantization, and a megakernel. Quote: Correspondence line, "Correspondence to: Song Han."

Jiaming Tang, Zhuoyang Zhang, Junxian Guo — MIT (Guo's work partly at NVIDIA; Tang's partly at Inco AI), equal-contribution first authors. They built PEAR and ran the RURL experiments. Quote: "*Equal contribution" (author footnote).

Zhijian Liu — UCSD / NVIDIA-affiliated co-author and senior collaborator on the efficient-systems side. Listed among authors (UCSD, Inco AI affiliation).

Phillip Isola — MIT, cited for the "Robot-use agents" framing the paper builds on. Quote: "Robot-use agents (RUAs) have recently become a promising direction for general-purpose robot control (Isola, 2026...)" (Section 1).

Tianxing Chen and the RoboDojo team — Authors of the RoboDojo benchmark used for training and sim-to-real. Quote: "RoboDojo (Chen et al., 2026) ranks robot agents in a separate track" (Section 2).

Xiaoman (Xinyao) Zhou et al., LIBERO-Pro authors — Created the benchmark that exposed the memorization weakness in VLAs. Quote: "LIBERO-Pro (Zhou et al., 2025) changes LIBERO's goals and layouts" (Section 1).

Kevin Black, Chelsea Finn, Sergey Levine and Physical Intelligence team — Authors of π0.5, the competitor baseline. Cited as Black et al., 2025a.

5. Operating Insights

Put robot-specific logic in the controller and interface, not the model

The portability result comes from a pose-level action space with each robot's own IK and controller underneath. The same protocol covers a single-arm Franka, a dual-arm ARX, and a dual-arm YAM, where "only the list of parts and the description of the robot change" (Section 3.1). The cost: "Accuracy therefore depends partly on the controller: a poorly implemented one can move the arm abruptly at some positions and miss the target pose" (Section 5). If you build on this pattern, controller quality and the embodiment description in the prompt become first-class engineering deliverables. The authors also admit that harder tasks "succeed only after we describe the embodiment and the task to it in detail in the prompt" (Section 5).

Budget latency, tokens, and context as the real constraints

Expect multi-second decisions and long contexts. Episodes average "about 94k tokens... 19.7 rounds and 39 to 59 images" (Appendix A), and real-robot successes used "between 23k and 132k tokens of context" (Section 4.4). Reasoning length is a tunable success-versus-speed knob (Section 4.6). On the full RoboDojo benchmark, "370 of the final model's 540 episodes (68.5%) end in failure because this window is exhausted" (Appendix E). A CTO should plan for context management, KV-cache economics, and a task profile that tolerates seconds-per-step. Local deployment is plausible, but "NVFP4 quantization reduces our model's accuracy and lengthens its chain of thought, so all evaluations use BF16 weights" (Section 4.6), so edge numbers don't directly reflect reported accuracy.

Evaluate generalization under goal change, not just in-distribution success

In-distribution LIBERO scores hid a huge difference between policies: π0.5 led by 22.5 points on LIBERO yet trailed by 8.8 on LIBERO-Pro (Section 4.2). If you are vetting a vendor's policy, ask for performance when the instruction changes in a familiar scene and when object layout changes, not only on the demo task distribution. Also watch the reverse: the RL model degrades on rewording (73.3% vs 95.8% for π0.5, Table 3), so measure both failure modes against your deployment's actual variability.

6. Overlooked Insights

Compute cost and absolute real-world reliability are modest relative to the headline

"Training took about 50,000 H100-equivalent GPU-hours in total" (Appendix A), a substantial bill for 340 RL steps and only a handful of tasks. The 60% real-world rate comes from just 30 trials on two task types (Section 4, Table 7), and failure patterns are task-specific: "No model picks up the air pump, which is knocked over in all 6 of its trials" (Appendix G). The full RoboDojo average is only 5.50% (Figure 11), with just 21 of 540 episodes succeeded, 15 of them on the two trained tasks. The generalization story is real on narrow tasks but is not yet broad capability; the authors acknowledge they "have not been able to scale up our training" and covered only "two simulators and a small set of tasks" (Section 5).

Cross-embodiment failures reveal a sensitivity profile worth noticing

Appendix C reports that the LIBERO-only model is most sensitive to the robot's initial state, "which costs it 36.5 points" on LIBERO-Plus (Figure 10c: 37.9% vs 74.4% baseline), while camera, lighting, and background shifts cost far less. Meanwhile the real-robot analysis shows the final model eliminated whole failure classes seen in weaker checkpoints: "The final model shows none of these last four kinds of failure" (unsafe pose, 32k-token reasoning blowups, partial lift, wrong object) (Appendix G). Taken together: RL training appears to buy robustness to perceptual perturbation and reasoning stability, while state-initialization dependence remains a gap, and the 32k-token reasoning cap is a real operational failure mode that RL training tamed, not one inherent to the approach.