How to Instruct Your… | arXiv Physical AI Research Summary

Summary for Physical AI Investors and Operators

1. Key Themes

Reannotating Existing Data Is a Legitimate Scaling Lever — Not Just a Preprocessing Step

The core claim of this paper is that you don't need more robot demonstrations to improve your policy — you need better language paired with the ones you already have. The authors show that VLM-generated annotations applied to existing clips improve policy performance without collecting a single new demonstration. On RoboCasa, the best fixed annotation type (Physical Motion) raised success rate from 44% to 46%, and the learned instructor pushed that to 49% — within 3 points of a theoretical per-task oracle at 52%.

The economics are striking: "Re-annotating an existing dataset with richer descriptions costs orders of magnitude less than collecting new demonstrations: a single VLM call produces a multi-sentence caption in seconds at sub-cent cost and with no skilled human labor, while every new demonstration requires teleoperation time, dedicated hardware, and environment setup." (§1)

At 1M-clip scale on MolmoBot, the compute cost for one annotation pass is roughly $1,100 and 5×10¹⁹ FLOPs. Dense annotation then matches the unannotated baseline on two MolmoSpaces task families while using ~62% less training compute — saving ~1.3×10²⁰ FLOPs. (§4.4)

No Single Caption Style Wins Everywhere — and Picking the Wrong One Hurts

This is the paper's most operationally important finding. Prior work (RT-H, ECoT) each committed to one annotation style and called it done. DeMiAn shows that's a mistake. Four annotation types — physical motion, scene composition, arm pose, and segment-level reasoning — each win on different tasks. Choosing incorrectly can degrade performance below the unannotated baseline.

"SlideDishwasherRack jumps from 38% to 75% with Physical Motion... while Arm Pose and Reasoning are markedly less useful. On MolmoSpaces, Scene Composition adds 13 percentage points on Pick tasks, and the Reasoning aspect improves 8 percentage points over the baseline in the NextTo task." (§4.1)

The per-task oracle (always picking the best annotation type) reaches 52% average success rate — 8 points above the baseline and 6 points above the best single fixed type. That gap is not recoverable with a single fixed annotation strategy.

A Small Asynchronous Instructor Model Closes Most of the Oracle Gap at Zero Latency Cost

The practical deployment problem is: if the right annotation type varies per task, how do you select it at runtime without a human in the loop? The authors solve this with a 2B-parameter instructor model (Qwen3.5-2B) that looks at the initial scene image and task description and generates the appropriate annotation before action execution begins.

Crucially, this runs asynchronously — the robot starts acting immediately, and the annotation is injected at the next action-chunk boundary once generation completes (~1.87 seconds). "Async tracks sync within fractional points on SR (49.0% vs 49.5%) while injecting the instruction into a rollout already in progress." (§4.2, Table 3)

The instructor adds +3.8 percentage points over a random-aspect baseline using the same policy checkpoint, confirming the gain comes from learned selection, not just annotation presence. (§4.2)

Dense Annotation Improves Both VLA Post-Training and World-Model Mid-Training

The paper tests two representative policy architectures: a VLA built on openpi 0.5 (π₀) and a video-based world-action model (WAM) built on NVIDIA's Cosmos-Predict 2.5. Dense annotations improve both, across two distinct training stages.

For WAM mid-training on 50K human-egocentric EgoVerse clips — before any robot action data is introduced — dense annotation improves downstream RoboCasa performance. For VLA post-training on 1M MolmoBot clips, annotated policies reach higher MolmoSpaces success rates earlier and at higher peak. Both improvements hold even after annotation-generation FLOPs are charged to the compute budget. (§4.4, Figure 5)

This is significant because it means the lever applies whether you're training from internet-scale video or fine-tuning on robot teleoperation data.

Attention Visualization Confirms the Model Is Actually Reading the Annotations

The paper provides mechanistic evidence that dense annotations aren't just noise. Without annotations, the action expert's attention collapses onto the <bos> token — the task description is essentially ignored, used only as a positional anchor. With physical motion annotations, "attention instead distributes across linguistically grounded units... interacting objects ('oven door'), motion verbs not present in the task label ('push,' 'retracts'), and directional adverbs ('inward,' 'away')." (§4.1, Figure 4)

This is not a trivial finding. It means current VLA architectures are leaving language-conditioning capacity on the table that richer annotations can unlock — without architectural changes.

2. Contrarian Perspectives

More Demonstrations Is Not Always the Right Answer to the Scaling Problem

The dominant industry assumption is that scaling robot data collection — more hardware, more operators, more hours — is the path to better policies. DeMiAn challenges this directly. The authors show that reannotating 1M existing clips with a $1,100 VLM pass can match the performance of training on substantially more data.

"At 1M-clip MolmoBot scale, DeMiAn matches the no-annotation baseline on MolmoSpaces NextTo and Color with ~62% less compute, positioning dense re-annotation as a complementary, compute-efficient lever alongside demonstration scaling." (§4.4)

For companies spending millions on teleoperation infrastructure, this suggests there may be significant untapped value in existing corpora. The bottleneck may be annotation quality, not data volume.

Committing to One Caption Style — As Most VLA Research Does — Is Suboptimal by Design

The field has largely treated language annotation as a solved preprocessing problem: pick a format (step-level chain-of-thought, motion primitives, scene descriptions) and apply it uniformly. DeMiAn's results show this is architecturally leaving performance on the table.

"Any fixed-aspect deployment would underperform an oracle that picks per task." (§1)

The oracle gap is 6 percentage points above the best fixed aspect. On individual tasks, the difference is much larger — the gap between best and worst annotation type on SlideDishwasherRack is 37 percentage points (75% vs 38%). If you've standardized on one annotation style across your dataset, you have a structural ceiling on policy performance that more data won't fix.

World Models Pretrained on Human Video Benefit from Dense Language Too — Not Just Robot Data

The conventional framing of video-pretrained world models (like those from NVIDIA, Google DeepMind) is that scale of visual data is the key transfer mechanism. DeMiAn shows language density during that pretraining phase also matters — even when the video comes from human egocentric footage rather than robot teleoperation.

Mid-training DeMiAn WAM on EgoVerse 50K clips with dense annotations improves downstream RoboCasa success rate after action-head fine-tuning, even after charging annotation FLOPs. (§4.4, Figure 5A)

This implies that teams building foundation world models from human video are potentially underinvesting in the language channel during pretraining. The signal isn't just in the pixels.

3. Companies Identified

NVIDIA Description: AI and accelerated computing company; developer of Cosmos-Predict 2.5 video diffusion model and Cosmos Reason 1 vision-language encoder. Why relevant: DeMiAn WAM is built directly on NVIDIA's Cosmos-Predict 2.5 DiT backbone. The paper is co-authored by NVIDIA researchers and validates that dense annotation improves the performance of NVIDIA's Physical AI stack. NVIDIA's MolmoBot dataset (1M clips) and MolmoSpaces benchmark are primary evaluation environments. Quote: "DeMiAn WAM: We follow prior work that adapts video-generative world models for robot control by decoding actions from their learned spatiotemporal representations with a Cosmos-Predict 2.5 DiT backbone." (§4)

Physical Intelligence (π) Description: Robotics foundation model company founded by Sergey Levine, Chelsea Finn, and others; developer of π₀ and π₀.5 VLA models. Why relevant: DeMiAn VLA is built on openpi 0.5 (π₀.5), Physical Intelligence's open-source VLA. The paper directly extends and improves Physical Intelligence's architecture. Any team deploying π₀-based policies could apply DeMiAn's annotation pipeline on top of their existing data. Quote: "DeMiAn VLA: We use the open-source openpi 0.5 as the VLA backbone, a PaliGemma vision-language backbone paired with a flow-matching action expert." (§4)

Alibaba / Qwen Team Description: Chinese technology company; developer of Qwen3-VL-30B and Qwen3.5-2B language models. Why relevant: Two Qwen models are the operational backbone of DeMiAn: Qwen3-VL-30B-A3B-Instruct generates all four annotation types at ~$0.0011 per clip, and Qwen3.5-2B is the instructor model that selects annotation type at deployment. The cost efficiency of the entire system depends on Qwen's MoE architecture keeping active parameter counts low. Quote: "The labeling model is Qwen3-VL-30B-A3B-Instruct." (§3); "We use Qwen3.5-2B as the instructor model." (§4.2)

University of Washington / Allen Institute for AI (Yejin Choi affiliation) Description: Academic institutions associated with co-author Yejin Choi, a leading NLP researcher. Why relevant: Yejin Choi's involvement signals the paper's positioning at the intersection of large language models and physical AI — relevant for investors tracking how NLP talent is migrating into robotics research.

Meta / EgoVerse contributors Description: EgoVerse 50K is a human egocentric video dataset assembled across ~1,500 hours of in-person recording. Why relevant: DeMiAn applies its annotation pipeline to EgoVerse, improving downstream robot performance from human video pretraining. This positions egocentric video datasets as a viable source of robot training signal when paired with dense language. Quote: "EgoVerse 50K, for example, was assembled over ~1,500 hours of in-person recording across many participants before any policy training could begin." (§1)

4. People Identified

Bosung Kim Lab/Institution: UC San Diego / NVIDIA (equal contribution) Why notable: Lead author. UC San Diego is an increasingly important node in Physical AI research; this paper's dual UC San Diego / NVIDIA authorship reflects the tight industry-academia coupling in the current robotics research landscape. Quote: Co-first author on the core DeMiAn system design and experimental evaluation.

Prithviraj Ammanabrolu Lab/Institution: UC San Diego / NVIDIA Why notable: Senior author and a recognized researcher at the intersection of language grounding and embodied AI. His work on language-conditioned agents is directly foundational to DeMiAn's framing. Quote: Listed as corresponding author; affiliated with both UCSD and NVIDIA, signaling industry-integrated research.

Yejin Choi Lab/Institution: NVIDIA / University of Washington Why notable: One of the most cited NLP researchers in the world, known for commonsense reasoning and language model research. Her presence on a robotics paper is a signal that the language-for-robotics problem is now attracting top NLP talent — and that NVIDIA is actively recruiting it. Quote: Co-author; NVIDIA affiliation listed.

David Acuna, Jaehun Jung, Alex Trevithick, Brandon Cui Lab/Institution: NVIDIA Why notable: NVIDIA Research contributors. Their involvement reflects NVIDIA's investment in the full Physical AI stack — from Cosmos foundation models to annotation pipelines to policy training.

Ruiyi Wang Lab/Institution: UC San Diego / NVIDIA (equal contribution) Why notable: Co-first author alongside Bosung Kim. UC San Diego's robotics and AI research output has accelerated significantly, and this paper reflects that trajectory.

5. Operating Insights

Every Robot Dataset You Own Is Probably Underannotated — and That's Costing You Performance

If your training data is labeled with single-line task descriptions ("pick up the mug," "open the drawer"), your policy is learning from a fraction of the signal already present in those demonstrations. The pixels contain spatial relationships, contact transitions, arm configurations, and subgoal structure that a one-line label discards entirely.

DeMiAn shows you can recover much of that signal for ~$0.001 per clip using a hosted VLM — no new hardware, no additional teleoperation. At 1M clips, the total annotation cost is approximately $1,100–$4,400 for all four aspects. For any company sitting on tens of thousands of robot demonstrations, reannotation is likely the highest ROI investment available before the next data collection run. The paper demonstrates a 5 percentage point average success rate improvement on RoboCasa and 62% compute savings on specific MolmoSpaces tasks from this single intervention. (§4.4, Abstract)

Build Your Annotation Pipeline to Support Multiple Caption Types, Not One

If you're designing a data pipeline for VLA training today, don't standardize on a single annotation format. The paper's clearest practical finding is that the optimal annotation type varies by task family, with gaps of up to 37 percentage points between best and worst annotation on individual tasks (SlideDishwasherRack: 75% with Physical Motion vs. 38% baseline). (§4.1, Table 2)

A practical implementation: generate all four annotation types (physical motion, scene composition, arm pose, reasoning) at annotation time — the marginal cost is roughly 4× — and train your policy on a mixture. Then deploy a small instructor model (2B parameters is sufficient) that selects the appropriate type per task at inference. The async injection pattern described in the paper adds zero wall-clock latency. The instructor closes most of the oracle gap (49% vs. 52% oracle, vs. 44% baseline) using only the initial scene image and task description as inputs. (§4.2, Table 3)

6. Overlooked Insights

The Attention Collapse Finding Has Implications Beyond This Paper

The paper's visualization showing that baseline VLA policies collapse attention onto the <bos> token — effectively ignoring the task description — is buried in Section 4.1 as a qualitative illustration. It deserves more attention from practitioners.

"In the baseline policy, attention over prompt tokens is overwhelmingly concentrated on the <bos> token, with negligible weight on the task description that follows; the prompt is, in effect, being used as a position anchor rather than being read as language." (§4.1, Figure 4)

This implies that language conditioning in current VLA architectures is more fragile than commonly assumed. Policies may be learning to ignore low-information task labels and route decisions primarily through visual features. If this finding generalizes — and the mechanism is plausible — it means language generalization benchmarks for VLAs may be measuring something other than language understanding. Teams evaluating language generalization in their policies should probe attention patterns before drawing conclusions about language grounding.

The Instructor Abstention Mechanism Is a Deployable Safety Primitive

A detail in the instructor training procedure that receives no emphasis: "Tasks where every aspect underperforms the no-annotation baseline are assigned an empty target, teaching the instructor to abstain." (§3, §C.2)

This is more than a training trick. It means the instructor has a learned signal for when dense annotation is likely to hurt — and produces no annotation in those cases, falling back to the task-only baseline. For production deployments where annotation quality is uncertain (novel tasks, out-of-distribution scenes), this abstention mechanism provides a graceful degradation path. The instructor won't confidently inject a wrong annotation type; it can opt out. Teams building annotation-conditioned policies should consider this abstention behavior as a first-class design requirement, not an edge case.