RAPID: Robot Agentic Programming from Demonstrations

1. Key Themes

Automated Agentic Coding Loop for Robotics

RAPID successfully bridges the gap between software coding agents and physical robotics by automatically generating, verifying, and refining robot programs from a single visual human demonstration. Instead of requiring pre-specified success signals or human-engineered primitives, the system infers all necessary components for an agentic loop. As stated in Section I: "Our key idea is to use demonstrations as the programming interface for supplying the task- and motion-level information needed by the agentic coding loop."

Object-Centric Relational Program Representation

To ensure programs generalize beyond the exact demonstration scene, RAPID does not replay fixed motions. Instead, it represents action primitives as trajectory-optimization programs that achieve object-level effects (e.g., pushing, flipping) and composes them using relational constraints that adapt to runtime geometry. Section IV notes: "By expressing primitives and their composition relationally, RAPID preserves the invariant structure of the demonstrated strategy while adapting to changes in object appearance, pose, geometry, physical properties, and scene configuration."

High Performance on Contact-Rich Nonprehensile Manipulation

The system was tested on 8 challenging nonprehensile tasks (pushing, flipping, toppling, etc.) in simulation, achieving an average success rate of 75.9% across 50 novel test scenes per task. This significantly outperformed baseline coding agent approaches like CaP-Agent0, which only achieved 14.6% even when given ground-truth success signals and evaluation-time refinement access (Table II).

Validated Sim-to-Real Transfer

RAPID was successfully deployed on a real Franka Research 3 robot arm for all 8 nonprehensile tasks. The system achieved high real-world success rates (e.g., 0.9 on Task 1, 0.8 on Task 7) compared to CaP-Agent0, which struggled to generate reliable trajectories. Section VI-D states: "contact-rich nonprehensile behaviors optimized in simulation can be transferred directly to the real robot and executed effectively."

2. Contrarian Perspectives

Demonstrations are Programming Interfaces, Not Replay Targets

Most imitation learning approaches try to directly map demonstrations to low-level actions. RAPID argues that demonstrations should instead be used to bootstrap a programming and verification environment. Section I explains: "A demonstration directly shows a feasible way to accomplish a task and, together with a language description, conveys the intended outcome... a single demonstration bootstraps the task specification, manipulation primitives, and verification environment for closing the agentic coding loop."

Hand-Engineered Primitives are a Scalability Bottleneck

Many robotics companies rely on manually designed primitive libraries (pick, place, push). RAPID challenges this for complex manipulation, arguing that manual design cannot scale to the diversity of contact-rich interactions. Section I states: "Capturing this diversity with a fixed primitive library is difficult, making manual primitive design a substantial engineering burden. Nonprehensile manipulation therefore provides a natural testbed for studying whether a coding agent can construct reusable manipulation primitives from a single demonstration rather than relying on human engineering."

Structured Programs Outperform VLA Models on Robust Generalization

While Vision-Language-Action (VLA) models are heavily funded and hyped, RAPID shows that structured, object-centric programs can drastically outperform them on benchmarks requiring robust spatial generalization. On the LIBERO-Pro benchmark, VLA models like OpenVLA and π0 scored 0 across all settings, whereas RAPID achieved near-perfect or high success rates (Table III). The paper notes this is because VLA models often use "fixed world-frame offsets and predetermined timing, limiting spatial generalization" (Section VI-B).

3. Companies Identified

OpenAI

Description: AI research and deployment company. Why relevant: RAPID uses OpenAI's Codex as the underlying coding agent to generate and refine the robot programs. "We use Codex [1] powered by GPT-5.6 Sol with high reasoning effort as the coding agent" (Section VI-A).

NVIDIA

Description: GPU and accelerated computing company. Why relevant: Co-author David Hsu is affiliated with NVIDIA. Additionally, NVIDIA's Newton Physics Engine is referenced in the conclusion as a future tool for scaling up simulation capabilities, particularly for deformable objects (References).

Physical Intelligence

Description: Robotics foundation model company. Why relevant: Their VLA models, π0 and π0.5, are used as baselines in the LIBERO-Pro benchmark. RAPID significantly outperformed both, highlighting a competitive alternative to end-to-end VLA approaches for robust manipulation (Table III, References).

Franka

Description: Robotics manufacturer. Why relevant: The Franka Research 3 robot arm is the physical platform used for both simulation benchmarks and real-world deployment of RAPID (Section VI-A, Fig. 5).

Intel

Description: Technology manufacturer. Why relevant: The Intel RealSense L515 camera is used to capture RGB-D observations for the real-world experiments, feeding the real-to-sim reconstruction pipeline (Fig. 5).

4. People Identified

Yuyao Liu

Lab/Institution: MIT / National University of Singapore Why notable: Lead author and corresponding researcher. Driving the implementation of the RAPID framework, bridging vision, simulation, and agentic coding.

Jiayuan Mao

Lab/Institution: University of Pennsylvania Why notable: Equally advising author. Notable for work in structured reasoning and program synthesis for embodied AI.

David Hsu

Lab/Institution: National University of Singapore / NVIDIA Why notable: Equally advising author. Brings expertise from both academia and industry (NVIDIA), indicating potential strategic alignment with large-scale simulation and robotics platforms.

Leslie Pack Kaelbling & Tomás Lozano-Pérez

Lab/Institution: MIT Why notable: Equally advising authors. Both are foundational figures in robotics, particularly in task and motion planning (TAMP). Their involvement signals that RAPID is a serious evolution of classical structured planning integrated with modern LLMs.

5. Operating Insights

One-Time Programming Cost Enables Reusable, Fast Deployment

For robotics operators, the economics of RAPID are highly attractive. The agentic coding loop takes time to run, but the resulting program is frozen and highly reusable. Section VI-B states: "this construction takes on the order of tens of minutes per task. This cost is incurred once per task: the resulting strategy and primitives remain fixed across the 50 novel scenes, with semantic role binding taking only a few seconds per scene." This means high upfront compute cost translates to near-instantaneous deployment in novel environments.

Real2Sim Reconstruction is the Critical Enabler

The system's ability to verify programs relies entirely on its capacity to reconstruct the physical scene in simulation. RAPID chains together modern foundation models (VLMs for properties, SAM 3 for segmentation, SAM 3D for mesh reconstruction, FoundationPose for registration) to build a MuJoCo environment. CTOs should note that the quality of this real-to-sim pipeline directly dictates the quality of the generated program and the success of sim-to-real transfer.

6. Overlooked Insights

VLMs Used for Physical Property Estimation

Beyond just identifying objects, RAPID uses Vision-Language Models to estimate physical parameters like mass, friction, and restitution to populate the simulation. Section VI-D notes: "leveraging the VLM’s prior knowledge to estimate physical parameters such as mass, friction, and restitution, together with SAM 3D’s strong 3D reconstruction capability, helps keep the real-to-sim and sim-to-real gaps manageable." This is a highly practical use of VLMs for robotics that goes beyond standard visual grounding.

Scene Variant Generation Only Works with the Right Representation

An ablation study reveals that generating synthetic scene variants (SV) to test the program is crucial for RAPID (dropping average success from 75.9% to 53.2% without it). However, when SV was added to a baseline (CaP-Agent0) that uses standard Python programs instead of object-centric relational programs, it provided no benefit. Section VI-B explains: "without our object-centric relational program representation, CaP-Agent0 struggles to construct a generalizable program that can solve diverse generated variants without overfitting to individual scenes." This implies that data augmentation/variant testing is useless if the underlying policy representation isn't structurally generalizable.