ω-0: A Latent Predicti… | arXiv Physical AI Research Summary

1. Key Themes

Unified Whole-Body Control Replaces Decomposed Locomotion + Manipulation Pipelines

The core contribution of ω-0 is demonstrating that a single model can generate coordinated whole-body actions for concurrent loco-manipulation — where the robot must step, lean, balance, and manipulate simultaneously — rather than treating locomotion and manipulation as separate sequential phases. The paper argues that existing decompositions "become limiting in tasks where stable execution depends on simultaneous stepping, torso adjustment, reaching, and contact maintenance" (Section 1). The model directly predicts "controller-compatible whole-body action latents" that are executed by the SONIC low-level controller, achieving 81.8% success rate across 11 household tasks versus 44.5% for the next-best humanoid baseline (ψ-0) (Table 2). This is a nearly 2× improvement, suggesting that the industry-standard approach of decoupling navigation from arm control may be leaving enormous performance on the table.

Latent Predictive World Modeling Beats Video Generation for Robot Control

ω-0 challenges the prevailing trend of video-centered world action models (WAMs) that generate future video frames and then convert them into actions. Instead, it uses "compact future observation embeddings as a lightweight predictive objective" (Abstract) — essentially predicting what the scene will look like in latent space without reconstructing pixels. The paper explicitly states: "Rather than building a video-centered pipeline that first predicts future dynamics and then converts them into actions, we learn a latent predictive world-action representation in which future observation embeddings and whole-body action latents are trained together" (Section 1). The ablation confirms this: removing the future visual latent prediction branch drops success rate from 79.1% to 64.5% (Table 4), proving the world-modeling signal matters — but it doesn't need to be a video.

Human-to-Humanoid Data Pipeline via Simulation Replay

A major practical contribution is the pipeline for converting cheap human demonstration data (public videos, VR-captured motion) into robot-executable training supervision. The approach uses SONIC to "batch-replay the motion trajectories in simulation and record the corresponding robot states and whole-body action latents" (Section 5.1). Trajectories that "cannot be reliably executed by SONIC, such as highly dynamic or physically infeasible trajectories, are filtered out" (Section 3.3). This means the team can leverage internet-scale human video data while maintaining real-robot executability — a critical capability for any company trying to scale humanoid training data without proportional teleoperation costs.

ω-HOME: 40+ Hours of Real-World Humanoid Household Data

The team collected ω-HOME, containing "40.3 hours of demonstrations, 4,827 episodes, and 24 tasks recorded at 30 Hz" with "synchronized egocentric RGB videos, exocentric RGB videos, exocentric depth videos, whole-body SMPL motions, robot proprioceptive states, and whole-body action latents" (Section 4). This is one of the larger real-world humanoid household datasets publicly described, and the multimodal design (six synchronized modalities per frame) is notably richer than typical tabletop manipulation datasets. The paper shows that adding non-overlapping ω-HOME data during pre-training improves fine-tuned success rate from 81.8% to 82.4% (Table 3) — a modest but consistent gain suggesting the dataset captures useful whole-body priors.

Single Multi-Task Model Outperforms Specialized Baselines

All 11 tasks are executed by "a single unified ω-0 model, rather than by task-specific policies, task-specific action heads, or separate locomotion and manipulation modules" (Section 1). The model was trained on approximately 200 demonstrations per task (2,220 trajectories total) and evaluated across tabletop manipulation, cleaning, laundry, object transfer, and mobile manipulation. The gap is largest on long-horizon tasks requiring extended whole-body coordination — ω-0 achieves 90.3% task progress versus 61.0% for DiT4DiT (Table 2).

2. Contrarian Perspectives

Video Generation Is a Distraction for Robot Control

The most contrarian claim is that generating future video — the approach used by prominent WAMs like MotionWAM and DiT4DiT — is actively unhelpful for real-time humanoid control. The paper states: "improving pixel-level video fidelity does not necessarily translate into better control: for real-time humanoid execution, the policy mainly needs compact future information that is useful for choosing the next whole-body action" (Section 1). They further argue that "when action generation depends strongly on a predicted video trajectory, temporal inconsistencies in that trajectory can be amplified into abrupt transitions, hesitant motion, or unstable whole-body coordination" (Section 1). The ablation on the current-image encoder reinforces this: using the Wan video encoder (designed for temporally continuous video) for single-frame inputs produced "overly static or hesitant behavior" with only 63.6% success rate, versus 79.1% with V-JEPA (Table 4). The implication: companies investing heavily in video world models for robot control may be optimizing the wrong objective.

Decomposing Locomotion and Manipulation Is a Structural Bottleneck

Most humanoid robotics companies (Figure 01, Apptronik, Boston Dynamics) architect their systems with separate locomotion and manipulation controllers. The paper argues this is fundamentally limiting: "Such decompositions can be effective when the robot first moves to a location and then manipulates while mostly standing still. However, they become limiting in tasks where stable execution depends on simultaneous stepping, torso adjustment, reaching, and contact maintenance" (Section 1). The 2× performance gap over ψ-0 (which uses a decoupled AMO controller for lower body) provides empirical evidence. For companies building humanoids for household or warehouse tasks, this suggests the architecture choice of decoupled vs. unified control could be a decisive competitive factor.

Egocentric-Only Vision Is Insufficient for Whole-Body Tasks

The robotics industry has largely converged on egocentric cameras as the primary (and often only) visual input. ω-0's omni-view variant, which uses exocentric observations for locomotion-heavy tasks, consistently outperforms the ego-only variant (81.8% vs. 79.1% success rate, Table 2). The paper explains: "Egocentric observations are well aligned with execution-time perception, but they often provide a limited view of the robot's global body motion and displacement in the scene" (Section 6.4). This challenges the assumption that a single head-mounted camera is sufficient for deployment, suggesting that multi-camera setups or external infrastructure may be needed for reliable whole-body humanoid operation.

3. Companies Identified

NVIDIA, Description: GPU manufacturer and robotics foundation model developer (GR00T N1.7), Why relevant: GR00T N1.7 is used as a baseline and significantly underperforms ω-0 (22.7% vs. 81.8% success rate). The paper notes GR00T "does not provide the same RTC mechanism used in our method" (Section 6.2), suggesting NVIDIA's public inference pipeline lacks temporal continuity features critical for whole-body control. Quote: "GR00T-N1.7 (NVIDIA et al., 2025) is a general-purpose robot foundation model for manipulation and loco-manipulation."

Physical Intelligence (π), Description: VLA model company (π-0.5), Why relevant: π-0.5 is used as a baseline and achieves only 27.3% success rate, versus 81.8% for ω-0. The paper notes π-0.5's "released model is not directly configured for our humanoid action space" and required action head expansion (Section 6.2). Quote: "π-0.5 (Intelligence et al., 2025) is a generalist vision-language-action model for robot control."

Unitree, Description: Humanoid robot manufacturer (G1 platform), Why relevant: The G1 humanoid is the physical platform used for all real-world experiments. Quote: "All real-world robot demonstrations reported in this section are autonomously executed by the learned policy on the G1 humanoid" (Section 6.6).

Pico (ByteDance), Description: VR headset and tracker manufacturer, Why relevant: Pico hardware (4 Ultra headset, trackers) is used for teleoperation-based data collection. Quote: "The human operator wears a Pico VR headset and two foot-mounted Pico trackers to provide head motion and lower-body motion cues" (Section 4.3).

Stereolabs (ZED), Description: Depth camera manufacturer, Why relevant: ZED Mini and ZED depth cameras are used for egocentric and exocentric visual capture during both data collection and deployment. Quote: "we use a ZED Mini camera as the onboard egocentric camera" (Section 4.3).

Inspire Robotics, Description: Dexterous hand manufacturer, Why relevant: Inspire DexHands are the end-effectors used on the humanoid for manipulation tasks. Quote: "we equip the humanoid with Inspire DexHands for dexterous manipulation" (Section 4.3).

Alibaba (Qwen), Description: LLM/VLM provider, Why relevant: Qwen3-VL-2B-Instruct serves as the vision-language backbone for the whole-body action VLM in Stage 1. Quote: "we fine-tune Qwen3-VL-2B-Instruct (Bai et al., 2025) into a whole-body action VLM" (Section 3.2).

4. People Identified

Zhe Li, Lab/Institution: MARS Lab, Nanyang Technological University (NTU) / BAAI, Why notable: Project lead and corresponding contributor. Has published multiple papers on humanoid locomotion and control (cited as author on RoboMirror, Freestyle, and language-to-locomotion work). His research trajectory focuses on bridging human motion data and humanoid control — directly relevant to data scaling challenges in commercial humanoid robotics.

Jianfei Yang, Lab/Institution: MARS Lab, NTU, Why notable: Corresponding author. Leads the MARS Lab at NTU, which has been productive in humanoid whole-body control research. Quote: Correspondence at jianfei.yang@ntu.edu.sg.

Shanghang Zhang, Lab/Institution: Peking University (PKU), Why notable: Corresponding author with research focus on robot learning and embodied AI. Having a foot in both NTU and PKU ecosystems suggests access to both Southeast Asian and Chinese robotics talent and hardware pipelines. Quote: Correspondence at shanghang@pku.edu.cn.

Zhengyi Luo, Lab/Institution: Referenced via SONIC controller work, Why notable: Developer of SONIC, the low-level whole-body controller that ω-0 depends on entirely for execution. SONIC is described as scaling "natural humanoid whole-body tracking to large motion corpora" (Section 2.1). The entire ω-0 architecture is built on top of SONIC's action latent interface — making Luo's controller work a critical dependency. Quote: "SONIC (Luo et al., 2025) scales natural humanoid whole-body tracking to large motion corpora and provides an effective controller interface for real-world humanoid execution."

5. Operating Insights

The Action Latent Interface Is the Key Architectural Decision

ω-0 doesn't predict raw joint angles or end-effector poses. It predicts "controller-compatible whole-body action latents" — a 64-dimensional latent vector plus 2 hand command scalars (66 dimensions total, Appendix A.1) — that are directly executed by the SONIC controller. This means the high-level policy operates in the controller's native representation space, not in joint space or task space. For teams building humanoid stacks, this suggests that the interface between the learned policy and the low-level controller should be a learned latent space, not a hand-designed action parameterization. The paper shows this enables smooth receding-horizon control at 7+ Hz with 25-step action chunks, executing only 8 steps before re-planning (Appendix B.1).

RTC (Real-Time Chunking) Is Non-Optional for Smooth Deployment

The training-time RTC mechanism — where the model is trained to predict the remainder of an action chunk given a clean prefix from the previous chunk — has a large practical impact. Without it, success rate drops from 79.1% to 71.8% (Table 4). More importantly, the paper notes that without RTC, "the executed trajectories tend to contain more hesitation, discontinuity, and corrective motions" (Section 6.5). At deployment, they use overlap blending between consecutive chunks (Appendix B.2, Algorithm 1). Any team deploying diffusion-based action policies on real robots should implement this pattern: cache unexecuted actions from the previous chunk, use them as a warm-start prefix for the next denoising pass, and linearly blend at chunk boundaries.

Proprioceptive State Input Is Critical for Whole-Body Tasks

Removing robot state conditioning drops success rate from 79.1% to 60.9% — the single largest ablation impact (Table 4). The state vector is only 47 dimensions: body joint positions, hand joint positions, and torso orientation as a 6D rotation representation (Appendix A.1). Notably, they deliberately exclude IMU linear acceleration and angular velocity, using only orientation. This is a practical signal: for whole-body humanoid control, knowing where your joints are matters more than rich inertial sensing. Teams over-instrumenting their state representations with high-frequency IMU data may be adding noise without benefit.

6. Overlooked Insights

The Model Was Trained on Only ~200 Demonstrations Per Task

The entire real-world fine-tuning dataset is "approximately 200 demonstrations per task and 2220 trajectories in total" across 11 tasks (Section 5.1). This is a remarkably small dataset for achieving 81.8% success rate on 11 diverse household tasks with a single model. The heavy lifting comes from the staged pre-training pipeline: Stage 1 learns action-aware visual-language representations from public human video datasets (ARCTIC, Xperience-10M, Motion-X), and Stage 2 grounds these into robot-executable latents via SONIC simulation replay. This suggests that the bottleneck for humanoid deployment is not raw teleoperation hours but rather the quality of the pre-training pipeline that converts diverse human data into robot-actionable supervision.

Quaternion Double-Cover Problem Causes Training Instability

A small but practically important detail: the team converts IMU torso orientation from quaternions to a "continuous 6D rotation representation" because "directly using quaternions may introduce discontinuities and destabilize training" since "the same 3D rotation can be represented by two antipodal quaternions, i.e., q and −q" (Section 3.3). This is a well-known issue in robotics ML but is frequently overlooked by teams building their first humanoid policy. The same conversion is applied during deployment (Section 3.5). Any team using IMU orientation as policy input should implement this conversion or risk subtle training instabilities that manifest as erratic whole-body behavior — exactly the kind of bug that's extremely difficult to diagnose on hardware.