LADDER: A Long-Horizon… | arXiv Physical AI Research Summary

1. Key Themes

Long-Horizon Task Performance Collapse is the Core Bottleneck for Generalist Robot Policies

The paper's central finding is that current generalist manipulation policies (π0.5, Cosmos-3, Bridge-trained baselines) perform well on single pick-and-place actions but catastrophically fail when asked to complete multi-step tasks from a single instruction. On single-subgoal tasks, the low-level policy achieves 85.4% success in simulation and 66.7% on real WidowX hardware. But performance drops to 26.9% at 2 subgoals and 0.0% at 5 subgoals in simulation. In the real world, multi-subgoal success falls to 1.2% on WidowX and 5.6% on Franka. As the paper states: "when given multi-subgoal tasks (2 to 10 subgoals), the low-level policy collapses, falling to 5.2% to 15.8% in simulation (0.5% at 8 to 10 subgoals) and, in real-world experiments, to 1.2% on WidowX and 5.6% on Franka" (Section 1, Introduction). This is the single most important data point for anyone deploying robots in real environments where tasks are rarely single-step.

Orchestration Triples Multi-Subgoal Success but Remains Bounded by Low-Level Policy Errors

Adding a VLM-based orchestrator that decomposes instructions into sequential subgoals, verifies completion visually, and retries failures roughly triples success rates. The best learned-policy orchestration methods (VoLo and Proprioceptive verifier) achieve 25.0–25.3% overall, up from 11.0% for the raw low-level policy. But the paper is clear that the bottleneck shifts to physical execution, not planning: "This shifts the primary limitation of learned-policy systems to the low-level policy's capabilities, with failure to manipulate subgoal target objects emerging as the main bottleneck" (Section 5.2). VLMs decompose instructions accurately (up to 93.1% exact match) but struggle with scene state tracking (at most 70.8%), and even when planning is correct, the robot arm fails to execute individual subgoals.

Tool Selection Orchestration Dramatically Outperforms Learned-Policy Backends

The most striking result: systems that bypass learned policies entirely and instead call motion primitives or task-and-motion planners achieve far higher success. GPT-Policy (using GPT-6 Astra to select tool calls) achieves 62.0% overall success, more than double the best learned-policy system. PSP-Agent (introduced in this paper) achieves 40.1% using the same VLM as the learned-policy orchestrators but with motion primitive backends. The paper states: "The two best systems execute with tool-based primitives. GPT-Policy and PSP-Agent reach 62.0% and 40.1% success overall, against at most 29.6% for the learned-policy systems" (Section 6). This suggests that for long-horizon tasks, the architecture choice (tool selection vs. end-to-end learned policy) matters more than the quality of any individual component.

A Benchmark That Isolates Task Horizon as a Variable

LADDER is designed to isolate the effect of task horizon by holding the environment, objects, and skills constant while scaling from 1 to 10 subgoals. 138 of 145 tasks are "incremental"—longer tasks that extend shorter ones in the same scene. This is a meaningful design choice because existing benchmarks conflate horizon with scene complexity, object variation, and instruction phrasing. The paper notes: "Rather than inflating task horizon through unstructured scene changes, our benchmark structures long-horizon manipulation across eight spatial and logical task categories" (Section 1). The benchmark also demonstrates sim-to-real correlation: "we observe preserved rankings of the methods. This is an encouraging result for using LADDER as a reliable rank-measuring tool for long-horizon tasks" (Section 5.3).

2. Contrarian Perspectives

End-to-End Learned Policies Are Not the Right Architecture for Long-Horizon Tasks

The dominant narrative in Physical AI is that scaling end-to-end vision-language-action (VLA) models with more data and compute will solve generalist manipulation. This paper provides strong evidence that for multi-step tasks, end-to-end policies are fundamentally limited—not by planning or reasoning, but by physical execution reliability. Even with a privileged verifier that has ground-truth simulator state (an unrealistic upper bound), the learned policy backend only reaches 29.6% overall. The paper states: "Physical execution is the main bottleneck. VLMs decompose instructions accurately, but tasks often fail during execution even when given ground-truth subgoals" (Section 6). This challenges the assumption that bigger VLA models alone will close the long-horizon gap.

More Test-Time Compute (VLM Calls) Does Not Help Learned-Policy Systems

A common assumption is that more inference-time reasoning (more VLM calls, replanning, verification) will improve robot performance. The paper finds this is true for tool-selection systems but not for learned-policy systems: "VoLo makes more VLM calls than the Proprioceptive verifier but achieves similar success, while PSP-Agent and GPT-Policy pair higher call counts with higher success using motion primitives" (Section 5.1). The low-level policy consumes the time budget "moving aimlessly and trying to reach the target object," and "half of the failed episodes complete no subgoal after the first 28% of the budget" (Section 5.2). For learned-policy backends, the returns to test-time scaling are essentially flat.

Task-and-Motion Planning (TAMP) Alone Is Insufficient for Long-Horizon Tasks

TiPToP, a symbolic TAMP system, performs no better than the raw low-level policy (9.4% vs. 11.0% overall). The paper finds that "TiPToP fails to find a plan for 50.9% of its subgoals" and "no plan is found for 5.2% of its subgoals at one subgoal and 74.0% at five" (Section 5.2, Appendix B). This challenges the classical robotics view that symbolic planning with motion primitives is sufficient—open-loop TAMP without verification or recovery degrades rapidly as task complexity grows.

3. Companies Identified

NVIDIA

Description: GPU and AI computing company; also develops robot foundation models and simulation platforms.
Why relevant: NVIDIA's Cosmos-3 world model (released as a policy for DROID) is evaluated as a low-level policy baseline. It underperforms π0.5: "Cosmos-3 completes fewer subgoals than π0.5 at almost every horizon" (Appendix A, Table 5). NVIDIA is also referenced for cuRobo (used in TiPToP's trajectory planning). The paper's co-author Jonathan Tremblay is from NVIDIA.
Quote: "Cosmos-3 completes fewer subgoals than π0.5 at almost every horizon, so the orchestrators in the main text use π0.5" (Appendix A).

Physical Intelligence (π0.5)

Description: Robotics foundation model company (now part of Google/Alphabet), developers of π0 and π0.5 VLA models.
Why relevant: π0.5 is the primary low-level policy evaluated on the Franka/RoboLab-Long suite, fine-tuned on DROID. It represents the state-of-the-art in generalist manipulation policies. The paper shows it achieves 100% on single-subgoal RoboLab tasks but collapses to 0% at 5 subgoals.
Quote: "On single-subgoal tasks, the low-level policy reaches a similar success rate as the Proprioceptive verifier (85.4% vs. 80.2%). As task horizon grows, the low-level policy rapidly collapses to 26.9% at 2 subgoals and 0.0% at 5 subgoals" (Section 5.1).

Google DeepMind

Description: AI research lab developing Gemini models and Gemini Robotics models.
Why relevant: Gemini 3.7 Flash is the default VLM used for orchestration, verification, and planning across most systems. Gemini Robotics-ER 2 is used for real-robot verification. GPT-6 Astra (from OpenAI) is used by GPT-Policy, the best-performing system. The offline VLM benchmark (Table 4) shows Gemini 3.7 Flash leading in state tracking (70.8%) and overall mean accuracy (80.8%).
Quote: "Gemini 3.7 Flash also leads state tracking (70.8%), the hardest capability for most models, and serves as our default orchestrator VLM with the highest mean accuracy over the four capabilities (80.8%)" (Section 5.2).

OpenAI

Description: AI company developing GPT models.
Why relevant: GPT-6 Astra powers GPT-Policy, the best-performing system overall at 62.0% success. GPT-5.6 Sol ties Gemini 3.7 Flash for plan decomposition at 93.1%. The paper notes GPT-Policy "incurs significant time and cost budget" (Abstract).
Quote: "Tool selection orchestration with GPT-Policy on GPT-6 Astra significantly outperforms learned-policy methods, but incurs significant time and cost budget and succeeds only ~62%" (Abstract).

Alibaba (Qwen)

Description: AI company developing the Qwen family of vision-language models.
Why relevant: Qwen3-VL-2B serves as the vision-language backbone for the Bridge-trained low-level policy on WidowX. Qwen3-VL-8B and Qwen3.8-27B are evaluated in the offline VLM benchmark, with Qwen3-VL-8B achieving 91.0% on plan decomposition but only 9.3% on state tracking (all correct).
Quote: "Qwen3-VL-8B: Planning 91.0%, State tracking all correct 47.9%" (Table 4).

Anthropic

Description: AI company developing Claude models.
Why relevant: Claude Opus 5 is evaluated in the offline VLM benchmark, achieving 90.3% on plan decomposition and 61.0% on state tracking.
Quote: "Claude Opus 5: Planning 90.3%, State tracking all correct 61.0%" (Table 4).

4. People Identified

Petko Petkov

Lab/Institution: INSAIT, Sofia University "St. Kliment Ohridski"
Why notable: Lead author of LADDER benchmark. INSAIT is an emerging AI research institute in Eastern Europe. This work represents a significant contribution to robot evaluation methodology.

Jonathan Tremblay

Lab/Institution: NVIDIA
Why notable: Co-author with equal supervision role. NVIDIA's involvement signals industry relevance of the benchmark. Tremblay is also co-author on VoLo, one of the evaluated orchestration systems.

Danda Pani Paudel

Lab/Institution: INSAIT, Sofia University "St. Kliment Ohridski"
Why notable: Senior author with equal supervision. His group is producing systematic, rigorous evaluation work that challenges prevailing assumptions about end-to-end robot learning.

Luc Van Gool

Lab/Institution: INSAIT / ETH Zurich
Why notable: One of the most cited computer vision researchers globally. His involvement lends credibility to the benchmark methodology.

Kevin Black (referenced, not author)

Lab/Institution: Physical Intelligence
Why notable: Developer of π0 and π0.5, the primary low-level policies evaluated. These models represent the current frontier of generalist manipulation policies.

5. Operating Insights

For Multi-Step Tasks, Architecture Choice Matters More Than Policy Quality

If you are building a robot system that needs to execute tasks with more than 2-3 sequential steps, the data is clear: you should not rely on an end-to-end learned policy alone. The gap between tool-selection orchestration (62.0%) and learned-policy orchestration (25.3%) is larger than the gap between any learned-policy system and the raw low-level policy (11.0%). The paper shows that even with a perfect oracle verifier (privileged verifier with ground-truth simulator state), the learned policy backend caps at 29.6%. This means no amount of orchestration cleverness will fix a learned policy that cannot reliably execute individual subgoals. If your product involves multi-step manipulation, invest in reliable motion primitives with geometric verification rather than betting solely on a larger VLA model.

Visual Verification is Non-Negotiable for Long-Horizon Execution

Without verification, 32.6–43.4% of subgoals fail because the orchestrator "moves on before the subgoal is completed and never returns to it" (Section 5.2). Adding visual verification drops this failure mode to 3.4%. The paper tested three trigger mechanisms (periodic, proprioceptive, privileged) and three verification settings (none, VLM visual, simulator state). The proprioceptive verifier—using gripper state to detect attempt completion and a VLM to verify—provides the best practical tradeoff. For real-world deployment, this means you need: (1) a reliable signal that a manipulation attempt has ended (gripper opening after sustained close), and (2) a VLM-based visual check of whether the subgoal was actually achieved. Without both, your system will silently fail on intermediate steps.

Time Budget Allocation Should Front-Load Execution, Not Planning

The paper finds that "the Proprioceptive verifier completes 42.1% of all subgoals in the first half of the budget and 49.5% by the end, indicating that a longer budget would help little" (Section 5.1). Failed episodes stall early: "half of the failed episodes complete no subgoal after the first 28% of the budget" (Section 5.2). This means if your robot hasn't made meaningful progress in the first third of its time budget, it likely won't succeed at all. For production systems, this suggests implementing early termination heuristics and focusing engineering effort on making the first few subgoal attempts as reliable as possible, rather than extending time limits.

6. Overlooked Insights

VLMs Are Better at Planning Than at State Tracking—And This Gap Has Deployment Consequences

The offline VLM benchmark (Table 4) reveals that frontier VLMs achieve 93.1% on plan decomposition (breaking instructions into subgoals) but only 70.8% on scene state tracking (correctly identifying which subgoals are complete from a camera image). This 22-point gap means that even with perfect planning, visual verification will produce false positives or false negatives roughly 30% of the time. For a 5-subgoal task, the probability that all 5 verification checks are correct is approximately 0.708^5 ≈ 17.8%. This compounds: the longer the task, the more likely a verification error derails execution. This is a fundamental limitation of VLM-based verification that won't be solved by better planning—it requires either better scene understanding models or complementary geometric/sensor-based verification (as PSP-Agent uses with depth-based geometric checks).

Revisit Tasks Are the Hardest Category and May Represent a Practical Ceiling

Revisit tasks—where an object must be moved to a destination, then moved again, potentially returning to a previous location—resist all methods. Even GPT-Policy achieves only 43.5% on revisit tasks, and learned-policy systems average just 2.3%. The paper notes: "Revisit, whose tasks mostly have 6 or more subgoals, resists both: it needs intermediate transfers where execution errors compound" (Section 5.1). This has practical implications: any warehouse, assembly, or kitchen task that involves re-handling objects (e.g., "put the item in the box, then take it out and put it on the shelf") will be substantially harder than sorting tasks of equivalent length. Companies should be aware that their hardest use cases likely involve revisitation patterns, and current systems are nowhere near reliable for these.