OnlyAct: Extracting… | arXiv Physical AI Research Summary

1. Key Themes

Pretrained Video Foundation Models Already Encode Action Information

The paper's central finding is that a frozen, pretrained video model (V-JEPA 2) already contains more action information in its embeddings than purpose-built latent action models trained from scratch. As stated in Section 1: "when a pair of consecutive frames is encoded as a two-frame clip, even the frozen V-JEPA 2 embedding carries more action information than latent-action models trained on the same data, as measured by a linear probe (Table 1)." Specifically, frozen V-JEPA 2 achieves an average R² of 0.347 across 12 continuous-action datasets, compared to 0.211 for DreamDojo and 0.067 for AdaWorld (Table 1). After contrastive adaptation, OnlyAct raises this to 0.490. The implication is significant: the field has been spending compute learning action representations from scratch when the signal already exists in foundation model embeddings.

Disentangling Action from Scene Without Reconstruction Bottlenecks

Standard latent action models use a low-dimensional bottleneck to force the latent to encode only what a decoder cannot recover from the previous frame. The authors argue this creates a fundamental trade-off: "tightening the bottleneck improves separation between action and scene at the cost of expressiveness, while loosening it preserves nuance at the cost of scene leakage" (Section 1). OnlyAct replaces this with a contrastive objective using flow-guided scene-swapped positives and same-video hard negatives, achieving scene invariance (average LPIPS 0.304) comparable to bottlenecked methods (DreamDojo: 0.299, AdaWorld: 0.293) while being substantially more expressive (Table 2).

Cross-Domain Action Transfer from Unlabeled Video

OnlyAct demonstrates that actions learned from one set of videos transfer to entirely unseen domains. The encoder is adapted on 168K unlabeled frame pairs from nine video sources, then evaluated on 13 datasets — 10 of which were never seen during adaptation. On RoboCasa (unseen), R² improved from -0.368 (frozen V-JEPA 2) to 0.322 after adaptation; on Robosuite (unseen), from 0.448 to 0.695 (Table 1). The paper also shows cross-source action transfer: actions extracted from DOOM gameplay can drive Minecraft video generation, reproducing "turn right and then perform the attack action" in a different game engine (Figure 5).

Lightweight Real-Action Control via Small Labeled Datasets

Once the action space is shaped by unlabeled pretraining, mapping real robot/controller actions onto it requires minimal labeled data. For discrete-action domains, only 300 labeled fine-tuning clips (100 per action) are needed; for continuous actions like nuScenes driving, 100 clips suffice (Appendix B.5). On nuScenes, OnlyAct achieves a Follow score of 0.928 (Spearman correlation between commanded and generated forward motion) versus 0.722 for DreamDojo (Table 4), demonstrating that the pretrained action space transfers effectively to real-action fine-tuning.

Unified Evaluation Protocol for Action Representations

The paper introduces a three-axis evaluation framework — action expressiveness (linear probing), scene invariance (decoder reconstruction of future frame from embedding alone), and action-conditioned video generation quality (fidelity, transferability, controllability) — across robot manipulation, camera navigation, games, and driving. This is notable because the field has lacked a standardized way to compare action representations. The Transfer Dir metric (cosine similarity between optical-flow direction of source and generated video) is particularly important as it directly tests whether an action representation can carry motion to a new scene, unlike Self PSNR which can be gamed by scene leakage (Section 4.4).

2. Contrarian Perspectives

Stop Learning Action Spaces from Scratch

The dominant approach in the latent action literature (AdaWorld, DreamDojo, Genie, LAPA) is to train an action encoder from scratch using reconstruction objectives. OnlyAct argues this is the wrong starting point: "training the action space from scratch can prevent it from generalizing to unseen domains" and "latents can encode scene-specific cues such as appearance, object identity, or viewpoint rather than the action itself, making transfer difficult when scene or embodiment shifts" (Section 2). The evidence is stark: even the frozen V-JEPA 2 embedding (no adaptation at all) outperforms purpose-built latent action models on average R² (0.347 vs. 0.211 for DreamDojo). Most robotics companies building world models are investing in learning action representations from scratch — this paper suggests that's leaving performance on the table.

Reconstruction Bottlenecks Are a False Trade-Off

The standard architectural choice of using a bottleneck to separate action from scene is challenged directly. The paper argues this "trades expressiveness for separation" and that "these constraints come at the cost of reducing the resolution of the action space" (Section 2). OnlyAct achieves comparable scene invariance without any bottleneck, by instead reshaping the geometry of a pretrained embedding space through contrastive learning. This means the expressiveness-separation trade-off that practitioners have accepted as fundamental may be an artifact of the from-scratch reconstruction paradigm, not an inherent property of action representation learning.

Multimodal LLM Embeddings Are Not Automatically Better for Action

In Appendix A.3, the authors tested the same OnlyAct recipe on Qwen3-VL-Embedding-8B (a multimodal language model) instead of V-JEPA 2. The result: average probe R² of 0.142, far below V-JEPA 2's 0.490 and even below frozen V-JEPA 2's 0.347. This challenges the assumption that larger, multimodal models are always better backbones. The paper attributes V-JEPA 2's superiority to its pretraining objective being specifically about "how video content evolves across space and time" (Section 3.1), whereas general multimodal embeddings may not encode motion as effectively. For teams choosing foundation model backbones for robotics, video-specific pretraining matters more than raw model scale or multimodality.

3. Companies Identified

Meta (V-JEPA 2)

Description: Meta developed V-JEPA 2, a self-supervised video foundation model (ViT-L, 300M parameters) pretrained on large-scale video.
Why relevant: V-JEPA 2 is the backbone of OnlyAct. The paper demonstrates that Meta's video foundation model already encodes substantial action information, making it a strategic asset for anyone building video world models. Meta's decision to release V-JEPA 2 under MIT license (Table 7) makes it accessible for commercial use.
Quote: "We instantiate the action encoder ϕθ with the encoder of V-JEPA 2 [4], a ViT-L foundation model pretrained on large-scale video. Because it is pretrained to model how video content evolves across space and time, the encoder represents motion, not only appearance, which makes it a natural starting point from which to extract action representations." (Section 3.1)

Google DeepMind (Genie, Veo)

Description: DeepMind develops interactive world models (Genie, Genie 2, Genie 3) and video generation models (Veo).
Why relevant: Genie is referenced as a key prior work in latent-action video world models [11]. DeepMind's approach uses latent actions learned from video, which is the paradigm OnlyAct improves upon. The competitive landscape for world model platforms includes DeepMind's Genie series.
Quote: "A growing line of work explores how they can be turned into video world models by conditioning generation on action signals [11, 46, 20, 19]." (Section 1)

NVIDIA (Cosmos)

Description: NVIDIA's Cosmos World Foundation Model Platform for Physical AI.
Why relevant: Referenced [1] as part of the action-conditioned world model landscape. NVIDIA is building physical AI infrastructure and world models, making action representation quality directly relevant to their platform.
Quote: Listed among methods that "assume explicit action or intervention supervision, such as controller inputs, robot states, camera trajectories, or action prompts [3, 63, 1, 56, 67, 23, 53, 27, 46, 20, 40, 36, 54, 61, 47]." (Section 2)

AgiBot (AgiBot World)

Description: AgiBot World Colosseo, a large-scale manipulation platform and dataset.
Why relevant: AgiBot World is one of nine training data sources for OnlyAct's contrastive adaptation (10% sampling weight, Appendix B.2). Licensed CC BY-NC-SA 4.0 (non-commercial).
Quote: "AgiBot World [2] 0.10" (Appendix B.2, training data sampling weights)

4. People Identified

Runjia Li, University of Oxford

Lab/Institution: University of Oxford (VGG/Visual Geometry Group)
Why notable: Lead author. Previously published VMem (consistent interactive video scene generation, ICCV 2025). Working at the intersection of video generation and world models.
Quote: "Rather than learning an action representation from scratch, we start from a pretrained foundation model, V-JEPA 2 [4], and extract actions from it with contrastive learning." (Section 1)

Philip Torr, University of Oxford

Lab/Institution: University of Oxford, AI4ER / Torr Vision Group
Why notable: One of the most cited computer vision researchers globally. Senior author on both OnlyAct and VMem. His group is building a coherent research program around interactive video world models.
Quote: Co-author on the paper; the work was "supported by the Clarendon Scholarship at the University of Oxford." (Acknowledgments)

Andrea Vedaldi, University of Oxford

Lab/Institution: University of Oxford, VGG (Visual Geometry Group)
Why notable: A foundational figure in computer vision (VGG image annotator, VGGFace). His involvement signals that Oxford's VGG is investing seriously in video world models for physical AI. Also co-author on VFMF (world modeling by forecasting vision foundation model features, referenced as [9]).
Quote: Co-author; the paper is hosted at runjiali-rl.github.io/onlyact/ with code release planned.

Tomas Jakab, University of Oxford

Lab/Institution: University of Oxford
Why notable: Co-author on VMem (ICCV 2025) and this work. Building a research thread on consistent, controllable video world models.
Quote: Co-author on the paper.

Mido Assran, Meta (V-JEPA 2)

Lab/Institution: Meta AI
Why notable: Lead author of V-JEPA 2 [4], the foundation model that OnlyAct builds upon. V-JEPA 2 is emerging as a critical infrastructure layer for video-based physical AI — this paper demonstrates an unanticipated use case (action extraction) that Meta's team may not have optimized for.
Quote: Referenced via [4]: "V-JEPA 2: Self-supervised video models enable understanding, prediction and planning."

5. Operating Insights

Use Pretrained Video Foundation Models as Action Encoders, Not Just Observation Encoders

For teams building world models or simulators, the standard practice has been to use pretrained models for encoding observations and learn action representations separately. This paper shows that the same pretrained model can serve as an action encoder by simply encoding frame pairs as two-frame clips. The adaptation is cheap: 7.1M trainable parameters (LoRA adapters, ~2.3% of the 300M encoder), 1.5 GPU-hours on a single RTX 4090, and 168K unlabeled frame pairs (Appendix B.1-B.2). This is dramatically cheaper than training a latent action model from scratch and produces more transferable representations. CTOs should evaluate whether their current action representation pipeline could be replaced by adapting a frozen video foundation model.

Evaluate Action Representations on Transfer, Not Just Reconstruction Fidelity

The paper introduces a critical distinction between Self PSNR (how well a world model reconstructs a video driven by its own inferred actions) and Transfer Dir (how well actions from one scene drive generation in a different scene). A representation that leaks scene content can achieve high Self PSNR by shortcutting to reconstruct the future frame, but fails on transfer. As the paper notes: "a representation that leaks scene content can reach a high Self PSNR by reconstructing the future frame but cannot carry its action to a new scene, Transfer Dir is the more direct test of an action representation" (Section 4.4). Teams evaluating world model quality should adopt transfer-based metrics, not just fidelity metrics, or they may be fooled by scene leakage masquerading as action understanding.

Map Real Actions to Latent Space with Minimal Labeled Data

For deployment scenarios where you need direct control with real robot actions, OnlyAct demonstrates that a very small labeled dataset (100-300 clips) is sufficient to learn an interface from real actions to the latent action space. The interface is simple: a lookup table for discrete actions (initialized at the mean latent action of labeled clips) or a two-layer MLP for continuous actions (Appendix B.5). After 800 fine-tuning updates, OnlyAct achieves Follow accuracy of 0.901 on DMLab and 0.928 on nuScenes (Table 4). This suggests that the expensive part of building controllable world models is the unlabeled pretraining, not the action alignment — which has implications for how teams should allocate data collection budgets.

6. Overlooked Insights

The Contrastive Training Data Is Surprisingly Small and Diverse

The entire action encoder adaptation uses only 168K frame pairs from nine sources, with Minecraft gameplay videos (VPT) being the largest single component at 25% sampling weight (Appendix B.2). This is a tiny dataset by modern standards. The fact that it generalizes to 10 unseen datasets suggests that the heavy lifting is done by V-JEPA 2's pretraining, not by the contrastive adaptation. For startups, this means the barrier to building a domain-specific action encoder is very low — you need a pretrained video model, ~168K unlabeled frame pairs from your domain, and 1.5 GPU-hours. The contrastive adaptation is essentially a fine-tuning step, not a training run.

Flow-Guided Scene Swapping Only Works for Static Cameras

A significant limitation buried in the method details: the flow-guided scene-swapping positive construction (the key mechanism for teaching scene invariance) only works when the camera is static. For moving cameras (egocentric, hand-held, driving), the method falls back to simple photometric augmentation, which "provides a weaker scene-invariance signal" (Appendix D). During training, only about 43% of samples use the scene-swapping path; on hand-held/ego-motion datasets, most samples fall back (Appendix B.2). This means OnlyAct's scene invariance is likely weaker for domains dominated by moving cameras — which includes most real-world robot teleoperation data and autonomous driving. The nuScenes results (Transfer Dir 0.659 vs. DreamDojo's 0.574, Table 3) suggest it still helps, but the mechanism is less principled for these domains. Teams working on ego-centric or driving world models should be aware that the core innovation may not fully apply to their data distribution.