Agent as Policy for… | arXiv Physical AI Research Summary

1. Key Themes

Zero-Shot Robot Control via Foundation Model Agents, No Task-Specific Training Required

The paper's central contribution is demonstrating that a general-purpose coding agent (GPT-6 Astra running in OpenAI's Codex CLI) can directly drive a physical robot arm through complex manipulation tasks — assembly, block construction, dice reorientation, targeted throwing, and bimanual towel folding — with zero task-specific or environment-specific training. The agent interprets visual observations, writes Python programs at runtime, computes motion targets, and issues commands through a robot interface, then adapts based on physical feedback. As stated in the abstract: "We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training." Success rates are strong: "AGP achieves success rates of 100%, 100%, and 80% on three block construction configurations" (Abstract) and "10 of 10 trials on dice flipping" (Section 4.2.1). This matters because it suggests a path to general-purpose robotic manipulation that bypasses the expensive data collection and model training pipelines currently dominating the VLA (vision-language-action) space.

Runtime Programming as the Control Mechanism

Rather than using a learned policy that maps observations to actions, AGP has the agent write and execute Python code during the task — computing object geometry from depth images, estimating grasp poses, triangulating positions from multiple camera views, and composing robot motion commands. The paper analyzes 37 retained Python working files across 18 sessions, with 29 focused on "perception and geometry" including "Depth measurement, color region selection, ray intersection, triangulation, surface fitting, and grasp offset estimation" (Table 8, Appendix E). The agent extends programs iteratively — for example, creating fit_head.py to estimate a held pin's pose from wrist depth, then extending it to compute corrected target orientations, with outputs directly feeding motion requests: "The resulting horizontal target is approximately (0.51325, −0.15820) m. The next motion request uses (0.5133, −0.1582) m" (Appendix E). This is a fundamentally different control paradigm from end-to-end learned policies.

Experience Accumulation and Strong-to-Weak Transfer

The paper shows that saved procedures, measurements, and scripts from prior executions improve efficiency on repeated tasks. For two-pair assembly, "task time decreasing by 29.3% from the first to the fifth execution" with "response latency decreases by 47.0%" (Section 4.3.1, Figure 2). More strategically, experience collected by an expensive model (GPT-6 Astra) can be transferred to a cheaper model (GPT-5.6 Terra): "Success increases from 1/5 with empty experience to 4/5 with Astra experience. Among successful trials, mean completion time and token usage decrease by 34.3% and 11.4%, respectively" (Section 4.3.2, Figure 3). This suggests a deployment strategy where expensive frontier models bootstrap task procedures that cheaper models then execute repeatedly.

Cross-Embodiment Generalization

Appendix G demonstrates AGP controlling a different hardware platform — a seven-joint P7 arm with a RealHand L6 dexterous hand (five fingers, six control channels) — performing a bottle grasp and lift. The agent "reorients the wrist, opens the thumb, and approaches the bottle through successive position adjustments" then "closes the fingers in stages and commands a 2 cm upward wrist motion" (Appendix G, Figure 12). While the bottle slips during holding, this provides "qualitative evidence that AGP can coordinate a different arm and dexterous hand through their available interfaces" — suggesting the approach is not locked to a specific embodiment.

2. Contrarian Perspectives

You May Not Need Trained Robot Policies at All

The dominant paradigm in Physical AI is training vision-language-action (VLA) models or diffusion policies on robot trajectory data. AGP challenges this directly: "AGP introduces an alternative to policies trained specifically for action generation by using a general purpose foundation multimodal large language model (MLLM) directly as the robot policy" (Section 2.2). The paper argues that the two existing agentic approaches — pre-generating programs (Code as Policies) and orchestrating learned skills (SayCan, RoboClaw) — both have fundamental limitations: generated programs "must express runtime decisions through explicit rules, making it difficult to anticipate how uncertain observations and unexpected outcomes should affect execution," while policy orchestration leaves "motion generation... limited by what the selected policies can perform" (Section 1). AGP instead keeps the agent in the full decision loop at runtime, handling perception, geometry, planning, and recovery. For companies investing heavily in VLA training pipelines, this suggests an alternative architecture that could shortcut the data collection bottleneck.

More Reasoning Effort Doesn't Always Help

The paper finds that GPT-6 Astra achieves identical 5/5 success on two-pair assembly across low, medium, and high thinking effort settings, with "limited benefit from additional thinking effort on this task" (Section 4.2.2, Table 2). Low effort actually used fewer tokens (3.69M vs 3.30M at high) at similar cost. This challenges the assumption that maximizing reasoning depth always improves physical task performance — suggesting that for well-structured manipulation tasks, the bottleneck may be observation quality and geometric computation rather than deeper chain-of-thought reasoning.

Conventional Pick-and-Place Is Too Easy to Be a Meaningful Benchmark

The authors explicitly state that standard manipulation benchmarks are insufficient: "We find that conventional pick and place tasks, such as placing an object into a bowl, pose little challenge for AGP, motivating more demanding tasks to evaluate its capabilities" (Section 4.1.2). This implies that many robotics demos showing pick-and-place may not meaningfully differentiate system capabilities — the real challenges are long-horizon reasoning, precision assembly with submillimeter clearances, dynamic actions like timed release during throwing, and deformable object manipulation.

3. Companies Identified

OpenAI

Description: Provider of GPT-6 Astra, GPT-5.6 Sol/Terra/Luna models and Codex CLI agent
Why relevant: OpenAI's frontier models serve as the core reasoning engine. The paper demonstrates these models can directly control physical robots. GPT-6 Astra is the default across all task groups. The strong-to-weak transfer experiment (Astra → Terra) directly models a cost-optimization strategy for OpenAI customers deploying robots.
Quote: "GPT-6 Astra (OpenAI, b) at high thinking effort is the default across all four task groups" (Section 4.1.1)

Anthropic

Description: Provider of Claude Code agent and Claude Opus 5 / Claude Fable 5.1 models
Why relevant: Claude Code is evaluated as an alternative agent system. Claude Opus 5 achieves 5/5 success on two-pair assembly (matching Astra) but at higher cost ($9.75 vs $4.47 per successful trial). Claude Fable 5.1 achieves only 3/5. This positions Anthropic as a viable alternative agent platform for robotic control, with different cost-performance tradeoffs.
Quote: "Within Claude Code, Fable achieves lower success than Opus and uses fewer tokens on successful trials, yet incurs greater time and cost" (Section 4.2.2)

I2RT

Description: Manufacturer of the YAM robot arm platform used in experiments
Why relevant: The YAM arm (six revolute joints, parallel gripper) is the primary hardware platform. I2RT provides the Python library for robot models, motor communication, and gravity compensation. The paper also tests a P7 arm with RealHand L6 dexterous hand (Appendix G), suggesting the approach generalizes beyond I2RT hardware.
Quote: "We use an I2RT YAM (I2RT) arm with six revolute joints and a parallel gripper in a tabletop workspace" (Section 4.1.1)

Intel

Description: Provider of RealSense depth cameras
Why relevant: The Intel RealSense D405 (wrist-mounted) provides aligned RGB and depth at 640×360 pixels, critical for the agent's geometric computations. A RealSense D435i and D435 are used in the cross-embodiment experiment (Appendix G).
Quote: "A wrist mounted Intel RealSense D405 provides aligned RGB and depth at 640 × 360 pixels" (Section 4.1.1)

Logitech

Description: Provider of the BRIO overhead camera
Why relevant: The fixed overhead Logitech BRIO provides rectified RGB at 1920×1080 pixels for scene observation. This is the primary scene-view camera the agent uses for planning.
Quote: "A fixed overhead Logitech BRIO provides rectified RGB at 1920×1080 pixels" (Section 4.1.1)

4. People Identified

Meng-Zhao Jia

Lab/Institution: University of Notre Dame
Why notable: Core contributor and corresponding author. Notre Dame's presence in Physical AI research, particularly in agent-based robot control, signals emerging academic capability outside the typical Stanford/CMU/Berkeley axis.
Quote: Co-authored the paper; contact mjia2@nd.edu

Yang Lin

Lab/Institution: University of California San Diego
Why notable: Core contributor. UCSD has growing strength in multimodal AI and robotics. Co-first authorship suggests significant contribution to the system design and experiments.
Quote: Listed as core contributor with equal contribution

Xixin Zhang

Lab/Institution: San Diego State University
Why notable: Core contributor. SDSU's involvement suggests this research is spreading beyond R1 institutions, potentially broadening the talent pool in Physical AI.
Quote: Listed as core contributor with equal contribution

Xiaobai Liu

Lab/Institution: San Diego State University
Why notable: Senior author at SDSU. His involvement in robot manipulation research, particularly the geometric and perception aspects, positions SDSU as an emerging node in Physical AI research.
Quote: Listed as author

Meng Jiang

Lab/Institution: University of Notre Dame
Why notable: Senior author. His group at Notre Dame is working at the intersection of language agents and physical systems, an increasingly important research direction as foundation models extend into embodied AI.
Quote: Listed as author; the paper emerges from his research group

5. Operating Insights

The Cost-Performance Tradeoff Is the Primary Deployment Barrier

The paper is transparent that execution time and inference cost remain substantial barriers. Dice flipping — a 10/10 success task — costs $21.07 per successful trial on average, taking 37.9 minutes (Table 1). Sequential towel folding costs $24.14 per trial at 50.8 minutes. Four-pair assembly costs $16.62 per trial. For any commercial deployment, these economics are prohibitive for high-volume tasks. However, the experience transfer experiment (Section 4.3.2) shows a viable cost-reduction path: use an expensive model (Astra) to bootstrap task procedures once, then deploy a cheaper model (Terra) for repeated execution, reducing success cost from $1.98/trial (Terra alone, 1/5 success) to effective $3.94/trial (Terra with Astra experience, 4/5 success). CTOs should evaluate whether their use cases have enough repetition volume to amortize the initial expensive-model exploration.

The Robot Interface Design Is the Critical Engineering Investment

AGP's success depends heavily on a well-designed robot interface ("bridge") that exposes calibrated observations, geometric queries, and motion commands. The interface provides seven commands (state, status, help, frames, move_ee, move_joints, gripper) with safety envelopes: "radial target envelope of 0.12 to 0.65 m, a grasp point height envelope of −0.050 to 0.60 m, and a maximum Cartesian step of 0.25 m" (Appendix A.1). Motion execution runs independently of agent inference, allowing the robot to execute while the agent reasons. This separation of reasoning from execution, combined with enforced workspace limits, is the safety architecture. Engineering teams building agent-controlled robots should prioritize this interface layer — it's where safety, calibration, and capability boundaries are defined.

Observation and Decision Time Dominate, Not Motion Time

The time breakdown in Table 6 (Appendix D.1) reveals that the "visual loop" (image capture, geometric queries, image processing, and model response) consumes the largest share of execution time across most tasks — 25.5 of 48.5 minutes for four-pair assembly, 16.2 of 38.6 minutes for dice flipping. "Action service" (actual robot motion) is typically only 20-30% of total time. This means optimizing robot speed or motion planning will yield diminishing returns compared to optimizing the agent's observation-to-decision pipeline. Reducing the number of observation rounds, improving image processing efficiency, and caching geometric computations are the highest-leverage optimizations.

6. Overlooked Insights

The Agent Writes Surprisingly Sophisticated Geometric Programs

Buried in Appendix E is evidence that the agent independently constructs non-trivial computational geometry programs — singular value decomposition for plane normal estimation, least-squares ray triangulation from two camera views, surface fitting from depth point clouds. The triangulate.py program converts "selected wrist and overhead pixels into calibrated rays in the robot base frame" and computes closest points using np.linalg.lstsq (Appendix E). The fit_head.py program samples point triples, identifies planar surfaces via SVD, intersects camera rays with the fitted plane, and computes corrected gripper targets. These are not pre-supplied libraries — the agent authors them during execution. This suggests that foundation models have latent geometric reasoning capabilities that emerge when given a robot interface and physical feedback, which has implications for how much hand-coded perception pipeline is actually needed.

Failed Recovery Reveals Hard Physical Constraints That Software Alone Cannot Solve

The six-block tower collapse case (Appendix H.1) is instructive: after successfully stacking all six cubes, the tower collapses during arm withdrawal. The agent then localizes scattered cubes but finds one beyond the robot's radial reach limit (0.70 m vs 0.65 m envelope), ending the attempt despite remaining budget. This illustrates that agent-based control cannot overcome fundamental hardware workspace limitations — reach, payload, gripper geometry. The paper notes "This case distinguishes a recoverable alignment error from a collapse that moves a required cube beyond reach" (Appendix H.1). For deployment planning, this means hardware selection (arm reach, workspace volume, gripper dexterity) remains a binding constraint even with intelligent software, and success criteria must account for post-action stability, not just task completion.