CLAP: Cross-Embodiment… | arXiv Physical AI Research Summary

1. Key Themes

Cross-Embodiment Video World Models Match or Beat Single-Robot Models

CLAP demonstrates that a single video world model trained across multiple robot embodiments (Franka, WidowX, bimanual YAM, G1 humanoid) can match or surpass models trained exclusively on one robot platform. On the DROID dataset, CLAP-CURR achieves a PSNR of 19.138 and LPIPS of 0.204, compared to the single-embodiment Ctrl-World baseline's 18.928 PSNR and 0.205 LPIPS (Table 4). The paper states: "CLAP's cross-embodiment models have the same number of model parameters and architecture as the single-embodiment video models... CLAP's cross-embodiment models match their single-embodiment counterparts despite utilizing effectively fewer DROID samples with the same training setup" (Section 4.1). This means you don't need a separate world model per robot — one model can serve your entire fleet.

Curriculum Learning Unlocks Internet-Scale Unlabeled Video

CLAP's core technical innovation is a two-stage curriculum: first learn physical priors from unlabeled video (including human videos) using latent actions, then ground those priors in end-effector action spaces for real-world deployment. The paper explains: "CLAP-CURR combines the unique strengths of latent actions with those of end-effector actions to overcome their individual limitations. This strategy achieves unrestricted cross-embodiment scaling without requiring downstream adaptation to target robot embodiments" (Section 3.1). This is significant because it means the billions of human manipulation videos on the internet become usable training data for robot world models — without needing action labels.

Zero-Shot Real-World Planning Improves SOTA Policies

CLAP serves as a "physical simulator in imagination" that can evaluate action proposals from multiple robot policies at inference time and select the best one. In real-world experiments on a Franka robot, CLAP improved π0.5's success rate on the lobster task from 90% to 95%, and MolmoAct-2's success rate on the tape task from 75% to 80% (Table 10). The paper notes: "CLAP addresses this core limitation by combining the strengths of multiple policies to compensate for their individual weaknesses" (Section 4.3). This means CLAP can be deployed as an inference-time planning layer on top of existing VLA policies — no retraining required.

Few-Shot Adaptation to Novel Morphologies (Bimanual, Humanoid)

CLAP's cross-embodiment pretraining provides a foundation that can be adapted to entirely new robot morphologies with minimal data. The paper demonstrates adaptation to a 14-DoF bimanual YAM robot and a 26-DoF G1 humanoid, achieving PSNR scores of 24.310 and 15.151 respectively (Table 14). Critically: "within a single gradient step, our adapted models generate meaningful video predictions, unlike models trained from raw SVD backbones" (Appendix D, Q21). This suggests that companies deploying new robot hardware could bootstrap world models dramatically faster by starting from CLAP's pretrained weights rather than training from scratch.

2. Contrarian Perspectives

Absolute Action Spaces Beat Relative Action Spaces for End-Effector Conditioning

The robotics community has largely assumed that relative action spaces (e.g., "move 3cm in x") are easier to learn than absolute action spaces because they have narrower distributions. CLAP's experiments contradict this for end-effector-conditioned video models: "relative-action spaces underperform absolute-action spaces in future prediction conditioned on end-effector actions across all perceptual metrics and robot environments, e.g., by about 14.6% in LPIPS in the DROID environment" (Section 4.2). The paper attributes this to compounding errors in relative actions over extended horizons. However, the finding reverses for language-conditioned models, where relative actions win due to tokenization constraints — a nuanced result that most practitioners would not predict.

Human Video Data Alone Is Insufficient — Robot Data Remains Essential

A popular thesis in Physical AI is that human video data (YouTube, egocentric video) can substitute for expensive robot demonstrations. CLAP directly tests this by training a model (DreamDojo-Human) exclusively on human videos using latent actions. The result: "CLAP's video models improve the DreamDojo-Human's LPIPS score by at least 61% on the DROID platform" (Section 4.1). The paper concludes: "although cross-embodiment video models can learn foundational physical priors from human video data, robot data is essential for effective transfer of these priors to robot morphologies" (Section 4.1). This challenges the narrative that human video alone will solve the data problem for robotics — robot teleoperation data remains a critical, non-substitutable input.

Cross-Embodiment Models Don't Need More Data or Compute to Match Single-Embodiment Models

Conventional wisdom would predict that a cross-embodiment model needs the union of all single-embodiment datasets plus more training steps to match a purpose-built single-robot model. CLAP shows otherwise: "to match the performance of single-embodiment baselines, cross-embodiment models typically require training on the union of all single-embodiment data alongside a significantly increased number of training steps. Surprisingly, CLAP's cross-embodiment models match their single-embodiment counterparts despite utilizing effectively fewer DROID samples with the same training setup" (Section 4.1). This implies that cross-embodiment pretraining learns transferable physical priors that make the model more sample-efficient, not less.

3. Companies Identified

Physical Intelligence (π0.5), VLA model developer, Used as a baseline policy that CLAP improves via inference-time planning. "We demonstrate CLAP's zero-shot generalization to real-world tasks, improving the performance of state-of-the-art robot policies, such as π0.5" (Abstract). CLAP boosted π0.5's success rate on the lobster task from 90% to 95% (Table 10).

MolmoAct-2 (Allen Institute / University of Washington), Action reasoning model, Used as a second baseline policy for cross-policy planning. CLAP improved MolmoAct-2's success rate on the tape task from 75% to 80% (Table 10). The paper notes MolmoAct-2 "struggles on the lobster task" while π0.5 "struggles on the tape task" — CLAP compensates for both weaknesses (Section 4.3).

NVIDIA (Cosmos), World foundation model platform, Referenced as a baseline (Cosmos-Predict 2.5) that underperformed the authors' internal Bridge model. "Although state-of-the-art baselines exist in the Bridge environment (e.g., WorldGym, Cosmos-Predict 2.5), we found these baselines to underperform our internal Bridge video models" (Section 4, Baselines). Also referenced for GR00T N1 humanoid foundation model (Reference [33]).

Stability AI / SVD (Stable Video Diffusion), Video generation backbone, CLAP uses SVD as its video backbone. The authors acknowledge SVD is older but chose it for cost: "newer backbones (such as Wan 2.2) yield higher-quality video, they require roughly 10 days of training per run on an H200 node compared to SVD's 2 days" (Appendix D, Q2).

Unitree (G1 Humanoid), Humanoid robot manufacturer, CLAP demonstrates few-shot adaptation to the G1 humanoid's 26-dimensional action space (7-DoF per arm, 6-DoF per hand). The adapted model achieves PSNR 15.151 on G1 video prediction (Table 14, Figure 14).

4. People Identified

Kechen Liu, Princeton University, Co-first author of CLAP. Equal contribution with Ola Shorinwa. The work emerges from the Princeton IRoM (Intelligent Robot Manipulation) lab ecosystem.

Ola Shorinwa, Princeton University, Co-first author. The acknowledgments thank "Prof. Anirudha Majumdar, Mingtong Zhang, and members of the Princeton IRoM lab" (Acknowledgments). Shorinwa appears on multiple related papers from this group including Playworld and uncertainty quantification for world models, suggesting an active research program around video world models for robotics.

Anirudha Majumdar, Princeton University (IRoM Lab), Acknowledged as a key collaborator. His lab appears to be producing a stream of related work (Playworld [26], LAP [30], uncertainty quantification [54], inference-time planning [52]), positioning Princeton as a hub for video world model research in robotics.

Chelsea Finn, Stanford University, Associated with multiple referenced works including Ctrl-World (the primary baseline), π0.5, OpenVLA, and RoboRewardBench. Her group's work is central to the experimental comparisons in this paper.

5. Operating Insights

Deploy CLAP as an Inference-Time Planning Layer, Not a Replacement for Existing Policies

The most immediately actionable finding is that CLAP can be layered on top of existing VLA policies (π0.5, MolmoAct-2, or your own) at inference time to improve real-world success rates — without retraining the base policy. The approach: sample k action proposals from each policy, use CLAP to predict future frames for each, score trajectories with a VLM reward model, and execute the highest-scoring trajectory. The paper shows this works even with CLAP-LAM (the weakest model variant) plus a lightweight adapter (Section 4.3). For a CTO, this means you can potentially improve manipulation success rates by 5-20 percentage points on hard tasks by adding a planning layer, at the cost of ~1.5-3 seconds of inference latency per decision (Appendix C.2).

Start World Model Development from Cross-Embodiment Pretraining, Not From Scratch

If you're building a video world model for your specific robot, CLAP's results strongly suggest you should start from a cross-embodiment pretrained model and few-shot adapt, rather than training from scratch or from a generic video backbone like SVD. The paper shows that "within a single gradient step, our adapted models generate meaningful video predictions, unlike models trained from raw SVD backbones" (Appendix D, Q21), and that all post-trained models surpass SOTA single-embodiment baselines after adaptation (Figure 11). Since CLAP open-sources all code and models, this is a ready-to-use starting point. The models run on consumer hardware (under 12 GB VRAM, fits on RTX 3060 per Appendix C.2), lowering the barrier to entry.

6. Overlooked Insights

CLAP Runs on Consumer Hardware but Is Far From Real-Time

Buried in Appendix D (Q4, Q5) and Appendix C.2: CLAP requires under 12 GB VRAM and fits on an RTX 3060, but inference takes 1.49 seconds on an H200 and 3.24 seconds on an A100 for 11 frames with 25 denoising steps. This is nowhere near the 10 Hz needed for closed-loop control. The practical implication: CLAP-style world models are currently useful for planning (where you evaluate a few candidate trajectories every few seconds) and for offline policy evaluation/finetuning, but cannot serve as real-time simulators in a closed-loop control pipeline. Companies betting on video world models for real-time control need to factor in this latency gap.

The RL Finetuning Result Is Small but Strategically Significant

In Appendix C.4, CLAP is used to finetune a diffusion policy via reinforcement learning inside the video world model — meaning the policy is improved by rolling out in CLAP's imagined futures rather than on real hardware or in a physics simulator. The improvement is modest (carrot task: 80% → 88% success, Table 15), but the implication is large: if video world models become reliable enough, they could replace physics simulators for RL-based policy improvement, eliminating the need for accurate URDF models, contact physics tuning, and domain randomization. The paper explicitly frames this: "CLAP achieves higher success rates... without degrading its performance... demonstrating CLAP's potential for cost-effective policy finetuning, circumventing the significant hardware and setup costs associated with alternative pipelines, such as real-world or simulation-based RL" (Appendix C.4).