Teahose.
SIGN IN
NEW HERE — WHAT TEAHOSE DOES
We read the entire AI & tech firehose — so you don't have to.
PODPodcastsAll-In, No Priors, Acquired…
NEWNewslettersStratechery, Newcomer…
PAPPapersPhysical AI research
PHProduct Huntdaily launches
VCInvestor ScoutSequoia, a16z, Benchmark…
CLAUDE DISTILLS →
7 reads, 30 sec each — free, 6 AM ET.
+ a live graph of the companies, people & themes underneath.
HOME/ARXIV PHYSICAL AI RESEARCH/One Prior, Two Dynamics: Repurpo…
PAPR
// RESEARCH PAPER
ARXIV PHYSICAL AI RESEARCH

One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning

DATE November 19, 2026SOURCE ARXIV PHYSICAL AI RESEARCHPARTICIPANTS YUXIN JIANG, MIKE ZHENG SHOU, ET AL. (ARXIV PHYSICAL AI)ARXIV 2611.26617
// KEY TAKEAWAYS5 ITEMS
  1. 01Actions Can Be Learned From Unlabeled Video Using a Video Generator as Both Teacher and Student
  2. 02State-of-the-Art Action Decodability With Almost No Labels
  3. 03Zero-Shot Transfer Across Embodiments
  4. 04Information Control, Not Architecture, Is What Makes Latent Actions Useful
  5. 05The Framework Generalizes Beyond Diffusion Priors
// SUMMARY

Bottom line: This paper shows that a single pretrained video generation model (Alibaba's Wan2.2) can be split into two halves — one that infers "what action caused this change" and one that predicts "what happens if I take this action" — and trained end-to-end on unlabeled video to produce a control representation that transfers from human hands to robot arms with zero robot data. For anyone building data pipelines for robots, this is evidence that the expensive part of robot learning (action labels) can be substantially replaced by video foundation models you don't have to train yourself.


1. Key Themes

Actions Can Be Learned From Unlabeled Video Using a Video Generator as Both Teacher and Student

The core contribution is architectural: instead of building an inverse dynamics model (IDM — infers actions from observed transitions) and forward dynamics model (FDM — predicts consequences) from scratch, O2LaF initializes both from the same pretrained video DiT. As the paper puts it: "Much of the prior needed for latent action learning therefore already exists inside a pretrained video generator" (Section 1). The IDM becomes "a single-pass latent action encoder" via learnable action queries; the FDM stays generative but is conditioned on the inferred actions (Section 3.3–3.4). The practical implication: the billions spent pretraining video generators are now a subsidy for robot learning.

State-of-the-Art Action Decodability With Almost No Labels

On robot manipulation, O2LaF achieves "an EEF R² of 0.277 on GR-1, compared with 0.149 for DreamDojo, despite training its encoder exclusively on human demonstrations without robot-domain adaptation" (Section 4.2). In plain terms: a model that never saw a single robot trajectory predicts robot end-effector motions from video nearly twice as well as a purpose-built robot world model (DreamDojo, from NVIDIA's GEAR team). On navigation, few-shot probing on MIND shows O2LaF "consistently outperforms prior latent-action representations... across labeled-data budgets" (Fig. 6, Section 4.2).

Zero-Shot Transfer Across Embodiments — Human Video Drives a Robot

The learned latent actions form a shared control space: "the same z inferred from 1st- and 3rd-person videos drives turn right in both navigation and robot scenes" (Figure 1). Quantitatively, human-to-robot and robot-to-human cycle reconstruction beats DreamDojo across all PSNR/SSIM/LPIPS metrics (Table 2), and the paper notes "O2LaF preserves arm-specific motion, including the active arm and reaching direction, despite substantial embodiment differences" (Section 4.3.2). This is the human-video-to-robot-policy thesis — the same bet behind companies building on egocentric human data — with the strongest published evidence to date.

Information Control, Not Architecture, Is What Makes Latent Actions Useful

The paper's systematic study (Section 5) shows the wins come from what each model is allowed to see: causal joint prediction "outperforms autoregression by removing clean-history shortcuts" (Section 1); training the FDM at high noise "suppresses this target leakage and forces the FDM to depend on the inferred action" (Section 3.4). The 5×5 noise sweep (Fig. 9) shows preferred noise configurations vary by domain — navigation peaks at (t_IDM, t_FDM) = (0.3, 0.98) while manipulation optima differ per arm. These are cheap, tunable knobs, not new architectures.

The Framework Generalizes Beyond Diffusion Priors

The same inverse-forward structure works with V-JEPA2 (Meta's self-supervised predictive encoder) as the IDM backbone, achieving "the strongest action probing performance" of all backbones tested (Table 4, Section 5.3) — though with a critical caveat covered below. This means the paradigm isn't locked to one video model family.


2. Contrarian Perspectives

Probing Benchmarks Are a Misleading Way to Evaluate Action Representations

Most companies evaluating latent action models use linear-probe accuracy against ground-truth actions. This paper directly undermines that practice: V-JEPA2 achieves the best probing scores (88.62/96.14 on MIND, Table 4) yet "its latent actions exhibit noticeable source-appearance leakage during cross-context generation" — the actions encode what the scene looked like, not just what the agent did. The paper's conclusion is blunt: "action probing alone does not reveal whether latent actions also encode task-irrelevant information that may hinder transfer. Downstream generative evaluation is therefore essential" (Section 5.3). If you're diligence-ing a company that quotes probing numbers, demand transfer results.

The Standard Diffusion Training Recipe Is Actively Harmful for Action Learning

The generative AI community's default timestep sampling (logit-normal, emphasizing intermediate noise — as in SD3) is exactly wrong here: "noisy future states may still retain sufficient information for the FDM to predict future states without fully utilizing latent actions" (Section 3.4). The paper deliberately trains at high noise, sacrificing generation quality for action quality, then recovers generation quality in a second stage (Section 5.4). This challenges the "just fine-tune the video model with the standard recipe" approach that VideoWorld 2 and CoLA-World take (Fig. 2, paradigms (c)).

Bigger Forward Models Improve Action Learning — Even With a Fixed Inverse Model

Counter to the intuition that action inference quality is bounded by the encoder, scaling the FDM from 1.3B to 14B "consistently improves action decodability with a fixed-size IDM" (Table 3b, Section 5.1). The supervisory signal quality — how well the forward model can predict consequences — directly shapes what the inverse model learns. This suggests compute spent on better world models pays off in action representations, an argument for investing in the FDM side that most action-learning teams ignore.


3. Companies Identified

Alibaba (Wan2.2 / Wan-Video) — Provider of the foundational video DiT (Wan2.2-TI2V-5B) that O2LaF is built on. Why relevant: the entire framework rides on an open-weights video generator, meaning any robotics company can replicate this stack. Quote: "Both the IDM and FDM of O2LaF are initialized from Wan2.2-TI2V-5B and adapted with rank-128 LoRA" (Section 4.1).

NVIDIA — Referenced extensively: DreamDojo (the main manipulation baseline, from the GEAR team), Cosmos world foundation models, GR00T N1 humanoid foundation model, and the GR-1 robot used for evaluation. Why relevant: O2LaF beats DreamDojo on its own benchmark using less training data ("O2LaF uses only EgoDex subset of DreamDojo's large scale egocentric training data," Section 4.1). NVIDIA's bet on robot world models is now competing with a cheaper, more transferable approach.

Google DeepMind — Genie 3 world models, RT-2, SIMA 2 embodied agent all cited as context. Why relevant: Genie-style latent action learning is the paradigm O2LaF claims to supersede; the from-scratch paradigm (Fig. 2a) includes "LAPA, Genie, AdaWorld."

Physical Intelligence — π0 VLA flow model cited (Section 1, ref [14]). Why relevant: π0 uses robot-supervised action experts; O2LaF proposes discovering the action space itself from action-free video — a potential pretraining stage for exactly these policies.

Meta — V-JEPA2 used as an alternative IDM backbone; Movie Gen cited among video DiTs. Why relevant: Meta's self-supervised predictive representations are validated as action-capable but flagged for appearance leakage (Table 4, Fig. 10).

ByteDance — Seedance 2.0 cited among large video generative models (ref [32]); villa-X (latent action modeling for VLAs, with ByteDance-affiliated authors) is a manipulation baseline.

Tencent — Three of the paper's authors (Wenqiang Sun, Jiaqi Li) are Tencent-affiliated, suggesting corporate research interest in this direction.

Anthropic — Referenced twice: "Frontier AI models are quickly becoming capable robot agents: they write robot policies as code and control a two-armed robot from camera images in simulation" (Appendix C.2), and Opus 5.5 is used to rebuild MuJoCo simulations from a single photo in the policy-verifier experiment. Why relevant: evidence that frontier LLM labs are being treated as robot-policy generators whose outputs need physical verification.

MiniMax, OpenAI — Cited as video generation / world simulator players (MiniMax H3; OpenAI's "Video generation models as world simulators").


4. People Identified

Mike Zheng Shou — Show Lab, National University of Singapore (senior/corresponding author). Why notable: leads one of the most productive labs in video generation and world models; also an author of Olaf-World, the direct predecessor this paper improves on. This is a coherent multi-year research program, not a one-off.

Lvmin Zhang — Stanford University. Why notable: the original author of the DiT architecture (Peebles & Zhang) that underpins all modern video diffusion transformers, plus recent work repurposing video DiTs for perception (TinyHistory). His involvement signals that the DiT community sees action learning as a first-class use of these models.

Ivor Tsang — ASTAR, Singapore. Why notable: senior figure in machine learning research; the ASTAR affiliation indicates Singapore's national AI program has skin in this game.

Yuxin Jiang — NUS Show Lab / A*STAR, first author, and also first author of Olaf-World (ICML 2026). Why notable: the individual driving the "video priors for latent actions" line of work; worth tracking for follow-ups on planning and closed-loop control.

Yann LeCun (via citations) — V-JEPA2 and LAWM-in-the-Wild co-author. Why relevant: his predictive-representation camp is directly benchmarked here, with the nuanced result that V-JEPA2 probes best but transfers worse than generative priors.

Shenyuan Gao (via citations) — Author of both AdaWorld and DreamDojo (NVIDIA GEAR ecosystem). Why notable: built the two strongest baselines O2LaF defeats; his trajectory marks the shift from from-scratch world models to video-prior-based ones.


5. Operating Insights

Don't Build Your Action Representation From Scratch — Repurpose and Adapt

The ablation is decisive: a frozen video DiT with only a trainable action head gets 43.88% (1st-person, k=1) on MIND, while rank-16 LoRA adaptation gets 70.27% — and rank-128 only marginally beats rank-16 (Table 3a). The message for engineering teams: you need to adapt the video prior, not just read features off it, but the adaptation is cheap (LoRA, not full fine-tuning). The full training run is modest by today's standards: 16 GPUs, 5k updates, ~65 GiB peak memory (Table 6) — this is a replicable recipe, not a moonshot.

Plan for Two-Stage Deployment From Day One

The end-to-end model learns good actions but generates mediocre video; the paper resolves this by "freezing the learned IDM and training a bidirectional FDM," which "substantially improves generation quality," and initializing that second-stage FDM from the jointly trained one "further improves both visual quality and action-following fidelity" (Section 5.4, Table 5). For a CTO: budget for two training phases — one that optimizes the action space, one that optimizes the world model consuming it — and reuse the first model as initialization for the second.

Diversify Your Unlabeled Video Across Domains and Embodiments

The data-composition study (Table 7) shows MiraData-only training gets 0.135/0.257 EEF R² on the robot, while EgoDexMix (human + robot manipulation video) reaches 0.227/0.386 — and also improves third-person navigation over human-only EgoDex. The paper concludes: "diversifying training videos across motion domains and embodiments can broaden the transferability of learned latent actions" (Appendix B.2). If you're curating video data for action pretraining, embodiment and motion diversity is a first-order design decision, not an afterthought.


6. Overlooked Insights

The Forward Model Is a Cheap Sim-to-Real Verifier — A Product in Itself

Appendix C.2 describes an underplayed application: use the FDM to predict what a candidate robot action will do in the real scene from a real photo, before execution. In their experiment, the physics simulator (rebuilt from a single head-camera photo by an LLM) predicted a successful salt-shaker grasp 18/20 times, while the FDM rollout from the real image predicted failure — because the rebuilt scene replaced "small tapered glass shakers with metal caps" with "taller plain cylinders" (Fig. 12c). The paper frames disagreement as "a flag worth checking before it reaches the robot." For anyone deploying LLM-written robot policies, this is a low-cost pre-execution sanity check that catches simulator-reality mismatches without a single real-world trial.

Latent Action Pretraining Is Also a Motion-Annotation Foundation Model

Appendix C.1 shows the IDM, trained with zero motion labels, transfers to supervised camera-motion recognition on CameraBench with large gains: "+44.9 and 9.2 AP for the spatial-temporal Transformer and pretrained video DiT, respectively" (Table 8) — and the 9.2 gain comes "on top of strong video generative pretraining." Even frozen IDM features are "highly effective for motion recognition." This suggests the same checkpoint gives you both a robot action space and a fine-grained motion annotator for labeling new video data — two data-flywheel components from one training run.

The Honest Limitation: These Actions Can't Plan Yet

The paper's own limitations section (Appendix D) is strategically important: "video diffusion models incur substantial computational costs and do not directly expose executable actions or task-relevant rewards. Bridging video-derived action representations with efficient, decision-relevant dynamics is an important direction toward planning and closed-loop control." O2LaF gives you a transferable action representation — the missing pieces for a deployable product are still fast rollout dynamics, executable action decoding, and reward signals. Companies claiming video world models alone will yield autonomous robots are ahead of what this evidence supports.