Decoding Task Progress… | arXiv Physical AI Research Summary

1. Key Themes

VLAs Already Know Where They Are in a Task — You Just Need to Ask

The central finding is that task progress (how far along a robot is toward completing a task) is already encoded inside the VLA's neural network in a form that a trivially simple model — a single linear equation — can read out. As stated in the abstract: "task progress, the normalized time remaining in a trajectory, is linearly readable from the activations." This isn't a deep learned decoder or a secondary neural network; it's a dot product. The embedding probe achieves R² = 0.807 on out-of-distribution evaluation, while a probe trained on raw observations collapses entirely (R² = −2627.7), confirming in Appendix B.4 that "task progress is therefore not linearly recoverable from raw observations; it is the VLA's representation that makes the feature decodable." This matters practically because it means the intelligence for self-monitoring is latent in the model you've already deployed — you don't need to build a separate perception system to track it.

This Capability Comes from Pre-Training, Not Fine-Tuning

A critical finding for anyone training or deploying custom VLAs: the progress signal was already present before any robot-specific fine-tuning. From Section 4.2: "the feature is observable in all four models, but the three π0.x variants substantially outperform PaliGemma and perform comparably to one another. This indicates that the progress feature is established by large-scale robotic pre-training and not by fine-tuning." The signal is present in the PaliGemma backbone (the vision-language foundation model) prior to any contact with robot data. This is a significant result for compute and architecture strategy — the generalization you need for runtime monitoring may already be baked into your foundation model.

A Free, Label-Free Failure Detector That Generalizes Across Task Types and Perturbation Modes

The paper deploys the progress probe as an out-of-distribution (OOD) detector that requires zero examples of failure during training. At the episode level across unseen perturbation modes, it achieves AUROC of 0.963 (Gaussian noise), 0.950 (color shift), 0.977 (blur), and 0.948 (occlusion), beating both supervised SAFE-MLP and SAFE-LSTM detectors in every mode (Table 7). Critically, this is achieved "despite never observing a single OOD example." The mechanism is straightforward: when the robot is stuck, its internal estimate of how far along it is stops decreasing. The detector flags this stall using the residual between predicted progress and expected progress defined in Equation 3. For operators, this translates to: deploy once, monitor across novel tasks and failure modes without re-labeling.

Fine-Tuned VLAs Are Ignoring Their Language Instructions at Runtime

Buried but alarming: the paper shows that after task-specific fine-tuning, π0.5 is largely ignoring the language prompt during execution. In Appendix A.1.3, the authors swap the target object name in the instruction entirely — "Put the pear..." becomes "Put the peach..." — and find that "in 10/10 episodes the policy still reaches for the original target and ignores the swapped word entirely." A position swap test shows the policy follows the memorized scene layout rather than the updated instruction, grasping the correctly relocated target in only 3/18 episodes (~17%). The progress probe, when trained with counterfactual language augmentation, can detect this failure mode — the probe output stops responding to language, serving as a diagnostic that "the model bases its actions on the memorized canonical scene layout rather than on the language instruction."

You Can Read the Model's State But You Cannot Rewrite It

Steerability — the ability to inject a modified representation and change the robot's behavior — fails entirely. Injecting a "boosted" progress signal at layer 0 does not propagate through subsequent layers (Figure 5, left), and the resulting actions are statistically indistinguishable from unsteered actions (Figure 5, right). The paper concludes in Section 5: "the progress feature τ is not steerable." This is a meaningful constraint: interpretability tools currently give you a window into the model but not a steering wheel. Any company claiming to "control" VLA behavior by editing activations should be treated skeptically given this evidence.

2. Contrarian Perspectives

Monitoring VLAs at Runtime Doesn't Require Separate Perception Models — and Building One is Wasteful

The prevailing approach to runtime failure detection in robotics is to train a separate anomaly detector, success classifier, or recovery policy that observes the same visual inputs as the main policy. This paper argues that approach is redundant. The VLA is already computing an implicit task progress estimate internally — one that is more generalizable than any external model trained on raw pixels. Appendix B.4 demonstrates this directly: a probe on raw observations achieves R² = −2627.7 on OOD data, while the embedding probe achieves R² = 0.807 on the same data. The paper makes the implication explicit: "VLAs already process the same visual and language inputs these external estimators consume... We hypothesize that progress is already encoded in the policy's own internal representations, recoverable without training a separate model at all" (Section 1). Companies building external monitoring stacks on top of deployed VLAs may be solving a problem that the model has already solved for them internally.

Supervised OOD Detectors Have a Hidden Generalization Tax That Makes Them Fragile in Production

SAFE-MLP, the strongest supervised baseline, achieves per-episode AUROC of 0.917 on unseen tasks — impressive. But it requires labeled OOD examples from specific failure modes to be collected in advance. In the cross-OOD generalization experiment (Table 2), Vτ — which never sees any OOD data — achieves 0.960 per-episode AUROC on held-out failure modes, beating SAFE-MLP (0.894) and SAFE-LSTM (0.857). The practical implication: supervised detectors are brittle to novel failure modes. In real-world deployment, the failure modes that matter most are precisely the ones you didn't anticipate. A detector grounded in the model's own internal progress estimate generalizes to failure types it was never trained on, because stalled progress is a universal signature of failure regardless of its cause.

The Interpretability Community Is Answering the Wrong Question for Robotics Deployment

Most VLA interpretability research focuses on cataloguing what signals exist inside a model — object positions, contact states, motion primitives. This paper explicitly redirects: "rather than cataloguing which signals are present in activations or steering the model directly, we target one specific signal, task progress, for runtime monitoring" (Section 2.1). The contrarian claim is that the operationally relevant interpretability question for robotics is not "what does the model know?" but "how is the model doing right now relative to its goal?" — a scalar, task-conditional, monotonically decreasing signal. This reframing has direct product implications: the first useful instrument panel for a deployed VLA is not a visualization of its world model, but a real-time progress gauge.

3. Companies Identified

Physical Intelligence (π)

Developer of π0 and π0.5, the primary VLA models used in all experiments.
Why relevant: π0.5 is the model being probed. All quantitative findings — progress decodability, OOD detection performance, steerability failure — are demonstrated on their architecture. The paper uses their publicly released fine-tuned checkpoints. Directly relevant to anyone licensing, fine-tuning, or deploying π0.5 in production. From Section 4.1: "The target model is π0.5, fine-tuned on VLABench."
Quote: "π0.5: a vision-language-action model with open-world generalization" (Reference [2])

Google DeepMind (PaliGemma)

Developer of PaliGemma, the vision-language backbone (400M SigLIP encoder + 2B Gemma) underlying both π0 and π0.5.
Why relevant: The progress signal is shown to exist in PaliGemma prior to any robotic fine-tuning. This has implications for any VLA built on PaliGemma or similar foundation models — the monitoring capability may transfer automatically. From Section 4.2: "the progress feature is established by large-scale robotic pre-training and not by fine-tuning."
Quote: "PaliGemma: A versatile 3B VLM for transfer" (Reference [11])

4. People Identified

Atiksh Bhardwaj & Edward Weiyi Duan (Equal Co-First Authors)

Lab/Institution: Cornell University
Why notable: Led the core experimental and conceptual work. Correspondence contact for the paper (Atiksh: ab2635@cornell.edu). The formalization of weak decodability, strong decodability, and steerability is their contribution, and it provides a transferable framework for evaluating any interpretability claim about VLAs.
Quote: "We present definitions for weak decodability, strong decodability, and steerability for generative models, separating the existence of a linearly correlated signal from its dependence on the correct inputs and its causal effect on outputs" (Section 1, Contribution 1).

Preston Culbertson

Lab/Institution: Cornell University
Why notable: Senior author and likely PI. Culbertson's group is building the interpretability and monitoring infrastructure for deployed VLAs, which is an increasingly important vertical as VLAs move to production. Worth tracking as a Cornell group to watch in the Physical AI safety and monitoring space.
Quote: Listed as corresponding senior author in the paper header.

Wei-Chiu Ma

Lab/Institution: Cornell University
Why notable: Co-author with likely expertise in visual representations and scene understanding. Relevant to the vision-grounding components of the decodability analysis.
Quote: Listed as co-author in the paper header.

Prithwish Dan

Lab/Institution: Cornell University
Why notable: Contributing researcher on the project; part of the Cornell VLA interpretability effort.
Quote: Listed as co-author in the paper header.

5. Operating Insights

Deploy a Progress Probe as Your First Runtime Monitor — It Costs Almost Nothing

A single linear probe (a weight vector and a bias term) trained on successful demonstrations — data you already have — gives you a real-time signal for whether your VLA is making progress toward task completion. No OOD labels required. No secondary model. The detector fires when progress stalls: "when the policy enters OOD states, it can no longer advance the task, so its progress estimate should stagnate" (Section 4.5). At the episode level, this approach achieves AUROC of 0.851 on unseen tasks and 0.960 on unseen failure modes (Tables 1 and 2). For engineering teams, the implementation is: (1) cache activations from layer 0 of the Gemma backbone during demo collection, (2) train a linear regression on normalized time-remaining labels, (3) at runtime, compare predicted progress to expected progress using Equation 3 and flag when the residual exceeds a threshold. The expected task completion time is estimated once from in-distribution rollouts. Total incremental compute at inference time is negligible.

Use the Probe as a Language-Grounding Diagnostic Before You Ship a Fine-Tuned Model

Before deploying a fine-tuned VLA, use the counterfactual probe to verify the model is actually conditioning on language. The test is cheap: run the model on the same visual observation with a swapped instruction and check whether the probe output changes. If it doesn't, the model has collapsed to scene-layout memorization and will fail on any instruction variant it hasn't seen before — exactly the finding in Appendix A.1.3 where 10/10 episodes ignored a swapped prompt. This is a pre-deployment red flag that is invisible to standard accuracy benchmarks. A model that achieves 95% task success but ignores instruction variation is not a generalizable policy; it's an expensive motion primitive. The contrastive probe makes this failure visible before it causes real-world incidents.

6. Overlooked Insights

The Progress Signal Generalizes Poorly Unless You Train on Diverse Tasks — and the Variance Is Huge

Figure 7 and Appendix B.1 contain a finding that should concern anyone training on narrow datasets: a progress probe trained on only 1 task generalizes to unseen tasks with R² ≈ 0.39 in the best case and ≈ 0.05 on average. The mean doesn't stabilize until 3–4 training tasks, and the variance across task combinations remains wide even at 8 tasks. The paper notes: "the specific tasks we include remain a strong source of variance" (Appendix B.1). For operators, this means a progress probe trained on a narrow manipulation curriculum may give you false confidence — it will look good on familiar tasks and silently fail on new ones. The practical prescription: ensure your demo dataset for probe training covers as many distinct task semantics as possible, not just trajectory volume. More diverse tasks, not more trajectories of the same task, is what drives generalization of the monitoring capability.

The Model Is Not Using Language to Control Its Actions — and Standard Benchmarks Won't Tell You

The language insensitivity finding (Appendix A.1.3) is presented as a diagnostic use case for the probe, but its implications are deeper than the paper foregrounds. The fine-tuned π0.5 model achieves competent task execution while completely ignoring the linguistic content of its instruction in 10/10 tested episodes. This means standard success-rate benchmarks — which test whether the robot completes the canonical task in the canonical scene — will not detect this failure. The model passes the benchmark by memorizing the scene, not by following the instruction. For investors evaluating VLA companies on benchmark performance, this is a significant caveat: a model that scores well on task completion may have zero generalization to instruction variants, novel object placements, or prompt rephrasings. The probe test described in this paper is a more discriminating evaluation than success rate alone, and should be part of any serious vendor evaluation protocol.