H2R-Bench: Benchmarkin… | arXiv Physical AI Research Summary

1. Key Themes

The Data Bottleneck for Robot Learning Is Real, and Video Generation Is Being Tested as a Solution

The paper opens with a crisp statement of the core problem facing the robotics industry: "collecting robot manipulation videos remains expensive, requiring specific hardware, teleoperation interfaces, calibrated cameras, safety constraints, and repeated physical execution." The proposed escape valve is using the vastly larger supply of human egocentric video — datasets like Ego4D and EgoDex contain thousands of hours — and using video generation models to synthetically translate human hand demonstrations into robot manipulation videos. This paper is the first systematic attempt to measure whether that translation actually works.

Video Generation Models Are Much Better at Looking Plausible Than Actually Transferring Manipulation

This is the paper's central finding, and it has direct implications for anyone planning to use generated video as robot training data. The benchmark measures five dimensions: task goal completion, action event completion, functional contact transfer, embodiment correctness, and video quality. The punchline: video quality scores cluster tightly between 0.73 and 0.81 across all 11 models, while the transfer-aware aggregate score (H2RCore) spans 30.0 to 84.6 — a 2.8x range. Rank correlation between the two is nearly zero: "H2RCore spans 30.0–84.6 and has weak rank association with M5 (ρ = 0.14)." HunyuanVideo 1.5-I2V achieves the highest video quality score of any model (0.806/0.808) but sits near the bottom of transfer quality with contact scores near 0.185 and embodiment scores near zero. A model can produce beautiful, temporally coherent video that is useless as robot training data.

Embodiment Consistency Is the Hardest Problem, Not Task Understanding

The most striking failure mode revealed by the benchmark is not that models fail to understand the task — goal completion and action completion scores are relatively high even for weak models — but that models routinely fail to actually replace the human actor with the correct robot. Veo 3.1, a top-tier proprietary model, achieves strong goal scores (0.725 for gripper, 0.715 for hand) but embodiment correctness scores of only 0.100 and 0.227 respectively. As the paper states: "Veo 3.1 attains the best Parallel-Jaw Gripper M1 score (0.725)... yet its M4 scores are only 0.100 and 0.227." The qualitative example is blunt: "Veo 3.1 produces a plausible interaction with the wrong end effector." This is a critical failure for any downstream policy training application.

Full Video Conditioning Dominates Frame Conditioning by a Wide Margin

The benchmark reveals a stark interface-driven performance gap. The three models that accept the full source video as conditioning input (Seedance 2.0, Wan2.7, Kling-V3) occupy the top three positions, with H2RCore scores of 74.5–84.6. The next best frame-conditioned model (Mitty-EPIC14B) scores 56.1–61.5, and most frame-conditioned models cluster between 30 and 50. A direct ablation on Seedance 2.0 quantifies the cost of degrading from video to image conditioning: "H2RCore falls by 27.6 points for the Parallel-Jaw Gripper and 41.4 points for the Dexterous Hand, while Video Quality changes by less than 0.02." Temporal context in the source video is not a luxury — it is load-bearing for functional transfer.

Dexterous Hands Are Systematically Easier to Generate Than Parallel-Jaw Grippers

Counterintuitively, dexterous hands — the more mechanically complex end effector — score higher across nearly every model. "Nine of the 11 models score higher with the Dexterous Hand, with an average H2RCore gain of 3.3 points." Contact transfer rises by 0.055 for all 11 models when switching from gripper to hand. The explanation offered is structural: "Its morphology is closer to the human actor in the source video, which is consistent with better preservation of contact during actor replacement." For parallel-jaw gripper generation specifically, "end-effector mismatch remains common." This has direct implications for companies choosing target embodiments and for teams trying to use generated data to train gripper-based systems.

2. Contrarian Perspectives

Conventional Benchmarks for Video Generation Are Actively Misleading for Robotics Applications

The standard approach to evaluating video generation — VBench, WorldModelBench, and similar — rewards visual fidelity, temporal consistency, and text alignment. The paper argues these metrics are not just incomplete but directionally wrong for robotics. Figure 4 makes this explicit: the model ranked highest on VBench-style video quality (HunyuanVideo) sits near the bottom on transfer quality. Most robotics and AI companies evaluating video generation models for data augmentation pipelines are using the wrong scorecard. The paper states directly: "Video Quality alone does not reliably recover the benchmark ranking across models." The implication is that procurement decisions, capability evaluations, and research directions based on general video quality benchmarks may be selecting for the wrong properties entirely.

Adding a Robot Reference Image Does Not Reliably Help — and Often Hurts

Intuition says: if you want a model to generate a specific robot, show it a picture of that robot. The benchmark tests this directly, and the results contradict the intuition for most models. "A target-robot reference image does not provide a uniform benefit." For Kling-V3, adding a reference image costs 13.5 H2RCore points for the gripper and 8.3 for the hand. For Seedance 2.0, it costs 8.8 and 4.4 points. Only Wan2.7 benefits substantially, gaining 6.6 H2RCore points for the gripper condition. The likely mechanism: visual reference images disrupt the model's ability to preserve source scene and interaction context. Teams building generation pipelines for robot data augmentation who assume reference images are a free improvement should test this assumption carefully.

The Benchmark Reveals That Task Recognition Is a Red Herring for Transfer Quality

A common assumption in the field is that if a model understands the task, it can transfer it. The data refutes this. Goal completion and action event completion scores are the most similar across models — even the weakest models often recognize what the task is. The separation between models comes entirely from contact transfer and embodiment correctness, which account for 60% of the H2RCore score by design. "A valid human-to-robot video should preserve the task intent, required actions, and functional interactions implied by the source demonstration" — but the paper shows these are separable capabilities, and most current models have the first without the second. Practical implication: a model that scores well on task understanding benchmarks may still produce training data with the wrong robot anatomy performing the wrong physical interactions.

3. Companies Identified

Seedance (ByteDance) Top-ranked video generation model in the benchmark. Relevant because Seedance 2.0 achieves H2RCore scores of 77.3 (gripper) and 84.6 (dexterous hand) — the highest of any evaluated model. "Seedance 2.0 ranks first with H2RCore scores of 77.3 for the Parallel-Jaw Gripper and 84.6 for the Dexterous Hand." Directly relevant to any company evaluating video generation models for robot training data synthesis pipelines.

Wan (Alibaba/Wanx) Two models evaluated: Wan2.7 (video-conditioned, second-ranked overall with scores of 76.5 and 83.1) and Wan2.2 (frame-conditioned, bottom tier). The gap between the two versions illustrates how much conditioning interface matters. Wan2.7 is also the one model where adding a robot reference image substantially helps: "contact transfer rises from 0.766 to 0.871 and embodiment correctness from 0.772 to 0.875." Relevant to teams evaluating open-weight generation models for embodied AI applications.

Kling (Kuaishou) Kling-V3 is the third-ranked video-conditioned model (74.5 and 81.7). Notable for being most sensitive to the reference image intervention — loses 13.5 H2RCore points when a robot reference image is added for the gripper condition. Relevant as a competitive data point for enterprise video generation procurement.

Google (Veo) Veo 3.1 is evaluated as a frame-conditioned model (first and last frames only). Despite strong goal completion scores, embodiment correctness is catastrophically low at 0.100 for the gripper. "Veo 3.1 produces a plausible interaction with the wrong end effector." Relevant because Google's Veo models are widely used and evaluated for robotics applications; this benchmark reveals a specific, serious failure mode.

xAI (Grok) Grok Imagine Video evaluated as a frame-conditioned model, scoring in the middle tier (50.1 gripper, 49.2 hand). One of two models that scores lower on the dexterous hand than the gripper, alongside Mitty-EPIC14B. Relevant to investors and operators evaluating xAI's video generation capabilities for physical AI applications.

HunyuanVideo (Tencent) HunyuanVideo 1.5-I2V achieves the highest video quality scores (0.806 and 0.808) of any model but ranks last or near-last on transfer quality (H2RCore 30.0 and 30.7). "HunyuanVideo 1.5-I2V achieves the highest M5 for both targets (0.806 and 0.808), but its contact score stays near 0.185 and its embodiment score near zero." The clearest example of the quality-transfer disconnect. Relevant as a cautionary case for teams using video quality metrics to select generation models.

Mitty / EPIC (referenced as Mitty-EPIC14B) Purpose-built human-to-robot video generation model, the only model in the benchmark specifically designed for the H2R task. Scores 61.5 (gripper) and 56.1 (hand) — competitive but not top. Has the lowest video quality score of any model yet substantially better transfer scores than most general-purpose models. "Mitty-EPIC14B has the lowest Video Quality score and substantially stronger overall scores." Directly relevant as a baseline for what task-specific model design achieves versus general-purpose video generation.

4. People Identified

Dingyi Rong, Shanghai Jiao Tong University / Shanghai AI Lab. First author. Benchmark design and evaluation framework construction. Key contact for understanding H2R-Bench methodology and future extensions.

Chaofan Ma, Shanghai Jiao Tong University / Shanghai AI Lab. Project lead. Drives the technical direction of the benchmark. Has broader work on spatial reasoning and segmentation that informs the evaluation design.

Yue Shi, Shanghai Jiao Tong University / Shanghai AI Lab. Corresponding author. Co-leads the project alongside Ning Liu and Guangtao Zhai. Has related work in 3D reconstruction and depth-aware neural rendering relevant to embodied AI.

Yao Mu, Shanghai Jiao Tong University / Shanghai AI Lab. Co-author with visible presence in the robotics policy learning space (cited in related work on 3D-aware policies, R3DP). Bridges the video generation and robot policy communities, making this work relevant to downstream policy training.

Guangtao Zhai, Shanghai Jiao Tong University. Corresponding author. Senior researcher with broad multimedia and video quality assessment expertise — directly relevant to the M5 metric design choices.

Ning Liu, Shanghai Jiao Tong University. Corresponding author. Provides senior oversight on the project direction.

Ryan Hoque et al. (EgoDex) — not authors of this paper but cited as the source of the 120 benchmark clips. "H2R-Bench uses 120 egocentric clips from the EgoDex test split (Hoque et al. 2025) as source videos." EgoDex is the foundational dataset enabling this benchmark, making its creators important to track for future data infrastructure in dexterous manipulation research.

5. Operating Insights

For Robot Data Pipelines, Evaluate Generation Models on Transfer Quality, Not Video Quality

If your team is building a synthetic data pipeline using video generation — translating human demonstrations into robot training data — the standard evaluation metrics used by the AI industry will steer you toward the wrong models. HunyuanVideo produces beautiful video that is nearly useless for robot training (embodiment score near zero). The paper's five-dimension framework — goal completion, action completion, contact transfer, embodiment correctness, video quality — is a practical rubric any engineering team can operationalize. The specific finding to internalize: "Video Quality alone does not reliably recover the benchmark ranking across models" (Spearman ρ = 0.14). Before committing to a generation model for your data pipeline, run it through at minimum an embodiment correctness check and a contact transfer check on your target tasks.

Video Conditioning Is a Hard Requirement for Usable Human-to-Robot Transfer

If you are selecting or building a generation model for cross-embodiment data synthesis, the interface type is not a secondary implementation detail — it is the primary driver of transfer quality. The paper's ablation on Seedance 2.0 shows that replacing the full source video with nine ordered frames drops H2RCore by 27.6 points for a gripper target and 41.4 points for a dexterous hand target. "Replacing the complete source video with nine images preserves much of the final-state and broad action evidence, but substantially degrades functional contact and embodiment realization." Models like Veo 3.1 and Grok Imagine Video, which only accept image frames, are structurally limited for this application regardless of their overall capability. When evaluating or procuring video generation infrastructure for robot data synthesis, full temporal video conditioning is a non-negotiable requirement.

Insertion and Assembly Tasks Are the Hardest to Transfer — Plan Accordingly

For teams working on precision manipulation (assembly, connector insertion, kitting), the per-task-family breakdown in Table S10 shows that F3 (insertion and assembly) is among the most challenging families for embodiment transfer. The interaction requires precise, localized functional contact — the exact failure mode that most models exhibit. Teams building manipulation systems for assembly-line or precision insertion tasks should not assume that a generation model performing well on rearrangement or mechanism actuation tasks will generalize. Task-specific evaluation on insertion scenarios is warranted before deploying synthetic data for these use cases.

6. Overlooked Insights

The Benchmark Evaluates Visual Evidence, Not Physical Executability — and That Gap Is Unquantified

Buried in the limitations section is a constraint with significant implications: "H2R-Bench evaluates visible evidence of human-to-robot transfer, not physical executability or downstream policy performance." This means a generated video could score perfectly on all five H2R-Bench dimensions — correct robot anatomy, correct contact region, correct task outcome — and still produce policy training data that fails when deployed on a real robot. The benchmark measures whether the video looks like a correct robot manipulation; it does not measure whether the robot trajectories implied by that video are kinematically feasible, collision-free, or learnable by a downstream policy. For investors evaluating companies claiming to use synthetic video generation to accelerate robot policy training, this gap between visual correctness and physical executability is the key unresolved question that H2R-Bench explicitly does not answer.

Tool Use Is Barely Harder Than Direct Manipulation — a Surprising Finding With Training Data Implications

Figure S3 in the appendix compares performance on direct manipulation (88 sources) versus tool use (32 sources) and finds almost no difference: mean H2RCore is 49.8 vs. 50.5 for the gripper and 53.1 vs. 53.7 for the hand. This is counterintuitive — tool use requires the robot to mediate contact through an object, which involves additional contact reasoning. The finding suggests that current models' failures are not primarily driven by the complexity of the interaction type but by the fundamental challenge of embodiment replacement and contact grounding regardless of tool involvement. For teams building training data pipelines, this means that investing in tool-use-specific generation improvements may be premature; the more fundamental problem of reliable embodiment transfer needs to be solved first across all interaction types.