World In-Context… | arXiv Physical AI Research Summary

1. Key Themes

Manipulation Skills Stored in Context, Not Weights

The paper introduces World In-Context Learning (WICL), a paradigm where a frozen foundation model learns a task by rehearsing in a simulator and writing the feedback into its context (prompt) rather than updating its network weights. As stated in the abstract: "We propose World In-Context Learning (WICL), a paradigm in which a frozen model starts from a demonstration, rehearses the task in a world such as a simulator, and writes the feedback into its context rather than its weights."

Outperforming Trained Policies with a Single Demonstration

RoboScript, the system implementing WICL, achieves 83.4% mean success across 50 tasks on the RoboTwin 2.0 benchmark using only one demonstration per task and no gradient updates. This exceeds the strongest trained action policy (4D-WAM, trained on 50 demonstrations per task) by over 20 points. The paper notes in Section 5.2: "With one demonstration per task and no gradient update, RoboScript (GPT-6.1-Sol) reaches a mean of 83.4%... The strongest trained policy, 4D-WAM trained on 50 demonstrations per task, reaches 61.7%."

Robustness to Visual Distribution Shift

Because the learned knowledge is semantic (recording object roles and task steps rather than visual appearance), it generalizes across visual distributions. The paper shows in Table 1 that RoboScript retains 99.5% of its clean success rate under domain randomization (textures, lighting, distractors), whereas the best trained policy (π0.5) retains only 65.1%. Section 5.3 explains: "Knowledge failures do not increase under randomization... The failures under randomization are physical, not epistemic, consistent with the semantic nature of the learned knowledge."

Cross-Embodiment Transfer to Real Hardware

Task memories learned in simulation on a simulated ALOHA robot were successfully transferred to a physical dual-arm PiPER robot with a different kinematic structure, without re-recording demonstrations or re-rehearsing. Section 5.7 states: "We take the frozen Task Memories from the main experiment (GPT-6.1-Sol) and apply a lightweight adapter that remaps the embodiment-specific parameters for the PiPER robot... The memories were rehearsed entirely in simulation on a different robot, yet they produce working behavior on real hardware through parameter remapping alone."

2. Contrarian Perspectives

Massive Datasets Are Not Necessary for High-Performance Manipulation

The dominant approach in robotics relies on collecting massive datasets to train Vision-Language-Action (VLA) models. This paper challenges that, showing that a single demonstration paired with simulator rehearsal is sufficient to outperform policies trained on 50 demonstrations. Section 5.2 states: "One demonstration with rehearsal is sufficient. With one demonstration per task and no gradient update, RoboScript (GPT-6.1-Sol) reaches a mean of 83.4%... exceeding every trained action policy."

Writing Skills into Network Weights is a Structural Limitation

Most robotics companies believe that the path to better robots is training larger models on more data. This paper argues that writing skills into weights structurally ties the model to the training distribution, causing catastrophic drops under domain shift. Section 1 notes: "FastWAM... reaches 77.8% mean success on clean scenes but drops to 1.9% under domain randomization... Every new task or distribution requires new data and new training."

Privileged Simulator Data Does Not Drive Real-World Performance

It is common practice to use privileged simulator state (ground-truth object poses, meshes) to train policies, which often leads to sim-to-real gaps. RoboScript enforces strict data isolation, and an ablation study shows that removing all privileged supervision during learning does not hurt performance. Section 5.4 reports: "This privilege-free variant reaches 86.5%, against 86.0% for the full method... the learned Task Memory does not depend on privileged information."

3. Companies Identified

Xiaomi: Referenced via "Xiaomi-Robotics-0" [8], a vision-language-action model baseline. Relevant as a competitor in the VLA space. Table 1 shows it achieves 62.9% clean success but drops to 18.2% under randomization.
Agibot: Referenced via "Abot-M0" [85], a VLA baseline. Relevant as a competitor. Table 1 shows it achieves 57.4% clean success and 30.4% randomized.
NVIDIA: Referenced via "GR00T-N1.7" [59], a VLA baseline. Relevant as a major industry player. Table 1 shows it achieves 43.6% clean success and 20.7% randomized.
Physical Intelligence (Implicitly via π0.5): Referenced via "π0.5" [2], a VLA baseline. Relevant as a leading VLA model. Table 1 shows it achieves 70.7% clean success and 46.0% randomized.
DeepSeek: Referenced as a foundation model provider ("DeepSeek-V4.1-Flash"). Relevant as a cost-effective model option. Table 1 shows RoboScript using DeepSeek achieves 77.8% clean and 78.4% randomized success.
OpenAI (Implicitly via GPT models): Referenced via "GPT-6.1-Sol" and "GPT-5.6-Luna". Relevant as the top-performing foundation model for the agentic approach. Table 1 shows GPT-6.1-Sol achieves 83.6% clean and 83.2% randomized success.

4. People Identified

Heng Fang: Huazhong University of Science and Technology / ArcheOne AI. Lead author. Developed the WICL paradigm and RoboScript infrastructure.
Xiang Bai: Huazhong University of Science and Technology / ArcheOne AI. Senior author. Notable figure in computer vision and robotics, overseeing the research direction.
Dingkang Liang: Huazhong University of Science and Technology / ArcheOne AI. Co-author. Involved in the system design and experiments.

5. Operating Insights

Treat the Simulator as the Robot's Terminal

Just as coding agents learn by running code in a terminal and reading the output, robotic agents can learn by rehearsing in a simulator and reading the feedback. This allows safe, cheap, and rapid iteration without risking real hardware. Section 1 states: "Coding agents show what is missing. They learn a new repository by running code in a terminal and reading what comes back... Now that foundation models have reached this capability, a simulator can serve the same role for a robot."

Enforce Strict Data Isolation for Deployability

To ensure that what is learned in simulation transfers to the real world, the system must enforce a strict boundary between deployable data (cameras, proprioception) and privileged simulator data. RoboScript uses a "memory linter" to reject any simulator-specific facts from the learned context. Section 4.2 explains: "A memory linter enforces this property by rejecting coordinates, ground-truth terms, and instance identifiers... This isolation ensures that the learned memory is not specific to any simulator."

Leverage Recursive Self-Improvement Without Retraining

Instead of collecting more data and retraining models, robots can improve by reflecting on their own evaluation records and writing guidance for future tasks. This allows continuous improvement with frozen weights. Section 4.4 states: "After an evaluation block, a reflection agent reads the episode records... It identifies failure patterns and their causes and writes a guidance document for the next block... Repeating this loop over several rounds lets the agent improve without gradient updates or privileged data."

6. Overlooked Insights

The Cost and Latency of Agentic Manipulation

While the performance is impressive, the agentic approach is expensive and slow compared to a trained policy. An evaluation episode with GPT-6.1-Sol takes on average 54.7 model calls, uses 2.33 million tokens, and runs for 149 simulation steps. Section 5.6 notes: "At list API prices, learning all 50 tasks and evaluating 1,000 episodes costs 308 USD with GPT-5.6-Luna and 1,911 USD with GPT-6.1-Sol." Furthermore, Section F acknowledges: "Inference is also slow compared with a trained policy, since an evaluation episode with GPT-6.1-Sol takes on average 54.7 model calls and the robot waits for each call before it acts."

The Importance of Geometric-First Perception

Rather than relying solely on foundation models for perception, RoboScript uses a geometric-first approach where object detection starts from the depth channel, and foundation models are only asked to resolve object identity. This makes perception more robust and reduces the burden on the VLM. Section 4.1 states: "Object detection starts from the depth channel. A height-map above the workspace produces connected-component proposals... Object identity is resolved by a three-way vote among Set-of-Mark annotation, a grounding model, and a VLM."