VIA: Visual Interface… | arXiv Physical AI Research Summary

1. Key Themes

Zero-Shot Robot Control via Visual Agentic Interfaces

VIA demonstrates that off-the-shelf coding agents (Claude Code, Codex) can control a robot manipulator through a browser-based 3D interface with no robot-specific fine-tuning and no access to privileged state information. The agent takes screenshots of a 3D point-cloud workspace, reasons about what it sees, and issues commands through a small set of MCP tools (move gripper, rotate, toggle, execute waypoint). The strongest configuration (CC-Fable) achieves 88% overall success across six diverse tabletop tasks and 96.7% on three LIBERO-Goal tasks with only a minimal prompt stating the goal. As the paper states in Section 1: "given the right interface, an existing agent can transfer its general perception, reasoning, tool-use, and UI-operation capabilities to robot control zero-shot, despite never seeing the interface before." This means the billions invested in general-purpose FM and agent capabilities (computer use, coding, visual reasoning) may already be transferable to robotics without the expensive robot-data collection pipelines that VLA approaches require.

Robot Control Rides the Scaling Curve for Free

A central finding is that VIA's performance improves directly with the strength of the underlying frontier model — no retraining required. Within the Claude Code family, upgrading from Opus 4.8 to Fable 5 improved success on every task with remaining headroom, raising overall success from 70% to 88% (Section 4.2, Table 2). On LIBERO-Goal, CC-Fable passed 29/30 seeds with only the minimal prompt, essentially matching the perfect score that CC-Opus achieved only with detailed instructions. The paper explicitly frames this as a strategic advantage: "VIA uses frontier models and agents unmodified, improvements in agent perception, reasoning, context management, or tool use transfers directly to robot control" (Section 1). For investors, this means a VIA-like approach gets better every time OpenAI or Anthropic releases a new model — the robotics capability is a free rider on the most well-funded R&D trajectory in AI.

Text Demonstrations Replace Teleoperation

VIA shows that providing an example waypoint list as plain text in the prompt can dramatically improve performance — CC-Opus went from 77% to 100% on LIBERO-Goal tasks when given detailed text prompts (Section 4.2, Table 2). This is significant because it suggests a path to robot "demonstrations" that doesn't require teleoperation hardware, recorded trajectories, or kinesthetic teaching. As the paper notes: "VIA offers a low-cost way to create 'demonstrations' for robots from plain text or screenshots, rather than teleoperation" (Section 4.2). This could lower the cost of data collection for robot learning by orders of magnitude.

Long-Horizon Planning Emerges from Agentic Reasoning

On the Rainbow task — arranging seven randomly placed colored blocks into a rainbow — CC-Fable achieved 100% success (Section 4.2, Table 2). This is a task that would require thousands of low-level steps for a typical learned policy like Diffusion Policy or a VLA. The paper highlights that the agent "always plans carefully before moving any block, estimating each block's location and choosing the ordering with the fewest moves" and adapts its strategy (left-to-right or right-to-left) based on initial block distribution (Section 4.2). This demonstrates that the long-horizon planning and re-planning capabilities of frontier agents — honed on software engineering tasks — transfer directly to sequential manipulation.

Cost Economics of Agentic Robot Control

The paper provides detailed cost analysis (Table 4). Per successful episode, costs range from $4.1 (Codex-5.6-Sol) to $15.1 (CC-Fable), with most tasks in the $2–$8 range. The Rainbow task (7 blocks, long-horizon) costs $23–$36 per success. While this is far too expensive and slow for real-time control, the paper identifies specific use cases where reliability matters more than speed: "assembly tasks where reliability matters more than speed, reset mechanisms for real-world reinforcement learning, and data collection in which a slow interface agent generates demonstrations for training fast policies" (Section 5, Limitations/Future Work). This positions VIA not as a replacement for fast policies, but as a reliable slow system that can bootstrap faster ones.

2. Contrarian Perspectives

Fine-Tuning on Robot Data May Be the Wrong Approach

The dominant paradigm in robotics — converting foundation models into VLAs by fine-tuning on robot data — is challenged head-on. The paper argues that fine-tuning degrades the very capabilities that make frontier models valuable: "fine-tuning on action data can degrade their reasoning and multimodal understanding abilities" (Section 2, citing Hancock et al., 2026). Furthermore, VLAs are "often orders of magnitude smaller than frontier FMs" due to compute constraints, which limits their general capability (Section 1). VIA sidesteps this entirely by using the frontier model as-is, through a visual interface. If this approach scales, it undermines the core thesis of companies building robot-specific foundation models — that you need massive robot datasets and specialized training to get capable robot policies.

Code-as-Policies Is Constrained by the Wrong Abstractions

The paper explicitly critiques the Code-as-Policies paradigm, arguing that it is "bottlenecked by human-crafted abstractions rather than by the FM" (Section 1). Citing Fu et al. (2026), the paper notes that CaP success rates "degrade sharply when the high-level primitives are replaced by low-level ones" (Section 1). VIA operates below even the lowest abstraction tier defined in that work: the agent commands generic 6-DoF waypoints and receives no perception primitives. This challenges companies building elaborate perception APIs and skill libraries for LLM-based robot control — the paper suggests that the model's own visual reasoning, accessed through a general interface, may be more capable and more scalable than any hand-crafted abstraction layer.

Slow, Expensive Inference Is Acceptable — and Possibly Preferable — for High-Value Tasks

Most robotics companies optimize for fast, cheap inference. VIA embraces the opposite: episodes take up to an hour of wall-clock time and cost dollars per success. Yet the paper argues this is a feature, not a bug, for certain applications. The key insight is that quasi-static tasks — assembly, data collection, environment reset — carry significant economic value and are not well-served by existing fast-but-unreliable policies. The paper states: "a wide range of quasi-static tasks carry significant value and existing methods are not yet reliable on them, making VIA a compelling option in this regime" (Section 5). This reframes the speed/reliability tradeoff: for a factory assembly step where a failure costs thousands of dollars in scrapped parts, spending $15 and 10 minutes to get it right the first time is a bargain.

3. Companies Identified

Anthropic

Description: AI lab behind Claude models and the Claude Code agent
Why relevant: Claude Code is one of two agents used to drive VIA. CC-Fable (Claude Fable 5) is the strongest configuration tested, achieving 88% overall and 96.7% on LIBERO-Goal. Anthropic also created the Model Context Protocol (MCP) that VIA's tools are built on. The paper references Anthropic's computer-use capabilities as foundational: "Frontier agents can now operate real software through native screenshots, clicking, and keyboard operations" (Section 1).
Relevant quote: "With the strongest model (Fable 5) it achieves 96.7% success on three LIBERO-Goal tasks and 100% on a long-horizon rainbow assembly task" (Abstract)

OpenAI

Description: AI lab behind GPT models and the Codex agent
Why relevant: Codex is the second agent used in VIA evaluation. Codex-5.6-Sol achieved 62% overall success and was the cheapest option at $4.1/episode. The paper references OpenAI's Operator computer-use product. GPT-5.6-Sol doubled success on the precision-demanding "Put bowl on plate" task from 40% to 80% over GPT-5.5 (Section 4.2).
Relevant quote: "Codex is much cheaper than CC: a successful episode costs $4.1 on average for Codex-5.6-Sol and $7.2 for Codex-5.5, versus $9.5 for CC-Opus and $15.1 for CC-Fable" (Section 4.2)

Physical Intelligence (π0)

Description: Robotics company building VLA models for general robot control
Why relevant: Referenced as a representative VLA approach (Black et al., 2025) that VIA positions itself against. The paper's critique of VLAs — that they are smaller than frontier FMs and that fine-tuning degrades reasoning — directly challenges the approach of companies like Physical Intelligence.
Relevant quote: "VLAs are often orders of magnitude smaller than frontier FMs given the limited data and compute available for fine-tuning, which in turn limits their general capability" (Abstract)

Google DeepMind (RT-2)

Description: Research lab that pioneered VLA models
Why relevant: RT-2 (Zitkovich et al., 2023) is cited as a foundational VLA approach. VIA's argument that fine-tuning degrades generalization directly challenges the RT-2 paradigm of converting web-trained VLMs into action models.
Relevant quote: "fine-tuning on action data can degrade their reasoning and multimodal understanding abilities" (Section 2)

4. People Identified

Dorsa Sadigh

Lab/Institution: Stanford University (senior author)
Why notable: One of the most influential researchers in robot learning, human-robot interaction, and foundation models for robotics. Her lab has produced foundational work on imitation learning, shared autonomy, and robot policy learning. She is also affiliated with SPHINX (referenced in Section 3.1), whose teleoperation interface VIA's UI is based on.
Relevant quote: The paper's core thesis — "your coding or computer-use agent is, in a sense, secretly a robot-control agent" (Abstract) — reflects her lab's focus on finding novel interfaces between foundation models and physical systems.

Priya Sundaresan

Lab/Institution: Stanford University
Why notable: Co-author of SPHINX (Sundaresan et al., 2025), the teleoperation interface that VIA's visual interface is directly borrowed from. Her work on salient-point-based imitation learning bridges teleoperation and policy learning, which is directly relevant to VIA's proposed use case of generating demonstrations for training fast policies.
Relevant quote: "This interface and waypoint-style control are largely borrowed from the teleoperation interface of SPHINX (Sundaresan et al., 2025), which demonstrated that humans can operate a similar interface to collect robot training data in both simulation and the real world" (Section 3.1)

Hengyuan Hu

Lab/Institution: Stanford University (lead author)
Why notable: Lead author of VIA, driving the conceptual leap from computer-use agents to robot control. The correspondence email (hengyuan.hhu@gmail.com) suggests he is the primary point of contact, indicating leadership of this research direction.

5. Operating Insights

The Interface Is the Moat, Not the Model

VIA's key innovation is not a new model or a new training method — it is a specific interface design that makes robot control legible to existing agents. The interface includes a 3D point-cloud workspace (chosen because it "resembles 3D design software (e.g., Blender) and 3D games that the model may have been trained on, so its 3D reasoning capabilities may transfer" — Section 3.1), raw camera feeds, and a small set of MCP tools designed around two principles: minimalism (each tool wraps a human operation with no extra abstraction) and agent ergonomics (discrete screenshot-based alternatives to continuous drag operations). For CTOs building robot systems, this means the interface layer between your FM and your robot is where engineering effort should be invested — not in fine-tuning. A well-designed interface that maps to capabilities frontier models already have (3D spatial reasoning, visual verification, sequential planning) will outperform a poorly-designed interface backed by a fine-tuned model.

Plan for a Two-Tier System: Slow Reliable Agent + Fast Trained Policy

The paper explicitly suggests VIA's most practical near-term use is as a data generation engine for training faster policies: "data collection in which a slow interface agent generates demonstrations for training fast policies" (Section 5). This implies a two-tier architecture: a slow, expensive, but highly reliable agentic system (VIA-like) that generates demonstration data or handles high-stakes steps, and a fast, cheap learned policy (diffusion policy, VLA) trained on that data for routine operations. Companies should evaluate whether their data collection pipeline could be replaced or augmented by an agentic system that requires no teleoperation hardware and can run autonomously. The text-demonstration finding (CC-Opus: 77% → 100% with text waypoints) further suggests that prompt engineering on the slow system can iteratively improve data quality without retraining.

Evaluate Models on Robot Tasks Without Robot Data — Use the Agent Benchmark

Since VIA performance scales with general model capability, companies can benchmark which frontier model to use for robot control without collecting any robot data. The paper shows clear scaling: CC-Opus (70%) → CC-Fable (88%), and Codex-5.5 (60%) → Codex-5.6-Sol (62%) (Table 2). The cost-performance tradeoff is also measurable: Codex is 2–3x cheaper but less capable; CC is more expensive but more reliable (Table 4). For a head of engineering, this means you can set up a VIA-like interface, test multiple off-the-shelf agents, and make a data-driven decision about which model to deploy — and switch models as new ones are released without any retraining.

6. Overlooked Insights

The Agent Can Improve Its Own Tools

Buried in Section 3.4 is a remarkable capability: because the agent controlling the robot is also a capable coding agent, it can suggest modifications to existing tools and implement new ones on its own. The authors describe spinning off runs where "an agent read the docstring of a newly defined tool and tested it in the environment" and then provided feedback on both functionality and documentation (Section 3.4). This means the tool layer is not static — it can co-evolve with the model's capabilities. For a robotics company, this suggests that the MCP tool library could be continuously refined by the same agents that use it, potentially reducing the engineering burden of maintaining perception and control APIs.

Precision Tasks Reveal a Fundamental Limitation of the Current Interface

The T-block task — building a T-shape with three blocks requiring ~8mm placement precision — was the hardest task for all agents (10–40% success, Table 2). All agents figured out the correct strategy (rotating the bottom block 45 degrees to increase contact area) but failed on execution precision. This reveals that the current interface's waypoint-based control with a simple PI controller is insufficient for precision-critical assembly. The paper acknowledges this: "a more sophisticated controller... could improve performance on precision-demanding tasks like T-block and help avoid collisions" (Section 5, Future Work). This is a critical signal for operators: VIA-style agentic control is ready for tasks where strategy and perception are the bottleneck, but not yet for tasks where sub-centimeter precision is required. Companies in precision assembly (electronics, aerospace) should view this as a near-term limitation with a clear engineering path forward.