FineVLA: Fine-Grained… | arXiv Physical AI Research Summary

Summary for Physical AI Investors and Operators

1. Key Themes

The "What vs. How" Gap Is a Real Deployment Blocker

Every robot dataset in production today tells a robot what to do ("pick up the cup") but not how to do it — which arm, which approach angle, where to grip, what to do if it slips. This paper quantifies the cost of that gap and shows a path to closing it. The authors unified 972,247 trajectories from 10 open-source datasets and found that nearly all are annotated with "a single goal-level description (e.g., 'pick up the cup') while the execution process—actor choice, approach direction, contact region, motion path, and state transitions—remains unspecified" (Section 1). The practical consequence: policies trained on this data cannot be steered by operators who need specific execution behavior.

Fine-Grained Supervision Doesn't Cost You Task Performance — It Gains You Both

The central counterintuitive result: adding detailed "how" instructions to training data doesn't hurt task completion rates. It improves them. "FG-only improves over Raw-only across all three (dataset, framework) combinations, with gains of +1.4/+2.0 on RDT-OFT, +7.0/+8.1 on RDT-GR00T, and +6.5/+4.7 on AlohaMix-OFT" (Section 4.3, Table 4). In real-world dual-arm manipulation, the best mixed model scored 62.7/100 vs. 49.9 for raw-instruction-only — a 26% relative improvement (Section 4.4, Table 5).

The Mixing Ratio Is a Tunable Knob, Not a Binary Choice

The paper identifies a precise training recipe: don't use only fine-grained instructions, and don't use only goal-level instructions — mix them. "Performance follows a consistent inverted-U trend, peaking around FG:Raw = 1:2 to 1:1" across all three architecture/dataset configurations tested (Section 4.3). This is an immediately actionable finding for anyone building VLA training pipelines. The optimal ratio of roughly 1:1 to 1:2 fine-grained to raw instruction held across simulation and physical hardware.

Steerability Is Measurable — and Currently Near Zero for Key Factors

The paper introduces a real-world benchmark that isolates whether a robot actually follows execution-level instructions. Under raw-only supervision, policies scored 24/100 on pose compliance and 22/100 on color compliance — effectively random compliance. Fine-grained supervision raised those to 47 and 40 respectively. "The largest gains appear on attributes invisible to goal-level language: Pose (+23), Color (+18), and Approach (+18)" (Section 4.4). This is the first rigorous quantification of how badly current training data fails at steerable control.

Open Infrastructure Exists Now to Retrofit Existing Datasets

FineVLA isn't just a research result — it's a toolkit. The authors release FineVLA-Tool (the pipeline to unify and annotate datasets), FineVLA-Data (47,159 human-verified trajectories), RoboFine-VLM (a specialized video annotator), and RoboFine-Bench (a benchmark for evaluating execution-level understanding). "We release the complete FineVLA suite—data pipeline, fine-grained annotations, benchmark, model checkpoints, and training code—to provide open foundations for steerable VLA research" (Section 7). Teams with existing robot data can apply this to their corpora without starting from scratch.

2. Contrarian Perspectives

More Descriptive Language in Training Doesn't Confuse the Policy — It Clarifies It

Conventional wisdom in robotics ML holds that overly verbose or complex language annotations introduce noise and distract from learning motor control. FineVLA directly contradicts this. The average instruction length increases 10.4× — from 9.3 words to 96.8 words per trajectory (Section 2.2, Table 1) — and yet policy performance improves across every tested configuration. The authors attribute this to dense language acting as explicit action constraints: "Raw instructions preserve compact goal semantics—what task should be completed—while fine-grained instructions expose execution constraints—how the task should be performed" (Section 5.2). The concern that verbose annotations degrade generalization is not supported by the data.

Architecture Choice Matters Less Than Supervision Quality

The robotics ML community invests heavily in architecture decisions — diffusion heads vs. flow matching vs. regression heads, dual-system designs, etc. This paper shows that the right language supervision nearly eliminates architecture-level performance gaps. "OFT is clearly stronger under Raw-only supervision (gap of 6.4/6.6 on Easy/Hard), but the gap shrinks as FG ratio increases and nearly vanishes under FG-only (0.8/0.5). This suggests that dense language supervision alleviates a supervision bottleneck, reducing the policy's dependence on decoder architecture choice" (Section 5.3). The implication: teams spending significant engineering resources on architectural novelty may be leaving a larger gain on the table by ignoring label quality.

The Annotation Bottleneck Is Solvable Without Human Reviewers at Scale

A common objection to fine-grained annotation is that it requires prohibitive human labor. The paper demonstrates a credible path around this: a robotics-specialized VLM (RoboFine-VLM) trained on 47,159 human-verified examples outperforms GPT-5.4 and Gemini-3.1-Pro on robotic video understanding — achieving 71.0% VQA accuracy vs. 62.1% for Gemini-3.1-Pro (Section 4.2, Table 2) — and can serve as a scalable annotator for future trajectories. "RoboFine-VLM is trained to support future scalable annotation and open-source reproducibility" (Section 2.4). The claim that fine-grained annotation is inherently human-bottlenecked is challenged by this result, though the authors acknowledge that human verification is still required to reduce annotation cost, not eliminate it (Section 5.5).

3. Companies Identified

Physical Intelligence (π₀.7) Developer of steerable generalist robot foundation models. Cited as a direct motivator for the research direction. "Recent robot foundation models such as π₀.₇, LingBot-VLA, GR00T N1.7, and GEN-1 suggest that future robot policies should not only infer what task to complete, but also follow instructions about how the task should be performed" (Section 1). FineVLA directly addresses the open data and infrastructure gap that closed-source systems like π₀.₇ face internally.

NVIDIA (GR00T N1.5 / Isaac GR00T) NVIDIA's GR00T architecture is used as one of the two action-decoding frameworks in FineVLA-Policy experiments. "StarVLA-GR00T adopts a dual-system design where the VL backbone serves as System 2 for slow reasoning and a DiT-based flow-matching module serves as System 1 for continuous action generation, consistent with GR00T N1.5" (Section 3.1). FineVLA's findings on fine-grained supervision apply directly to GR00T-based systems, suggesting NVIDIA's platform would benefit from this approach.

Alibaba / Qwen Team Alibaba's Qwen3.5-397B-A17B model is the backbone for RoboFine-VLM and the annotation pipeline. "We fine-tune Qwen3.5-397B-A17B on FineVLA-Data to obtain RoboFine-VLM" (Section 2.4). Alibaba researchers (Qwen Team members) are co-authors on this paper, making this a direct Alibaba research contribution with commercial implications for their model ecosystem.

OpenVLA / Open X-Embodiment Collaboration The OpenVLA-OFT architecture is one of the two policy frameworks tested. "StarVLA-OFT attaches a lightweight MLP regression head... following OpenVLA-OFT" (Section 3.1). The Open X-Embodiment dataset infrastructure underlies the source data. "Existing open-source robot datasets use diverse action and state representations that cannot be directly unified" (Section 1) — FineVLA's canonicalization tool directly addresses this limitation.

Generalist AI (GEN-1) Listed alongside π₀.₇ and GR00T as motivation for steerable robot foundation models requiring execution-level instruction following. "However, the data construction and evaluation infrastructure behind many such systems remains limited or closed. FineVLA complements these efforts by providing an open action-aligned annotation pipeline" (Section 6).

4. People Identified

Tao Yu Corresponding author; XLANG Lab, University of Hong Kong. Lab director and senior researcher behind FineVLA. Yu leads XLANG Lab, which has produced multiple influential works in language-grounded robotics and code/agent systems. His lab's focus on open-source infrastructure for embodied AI makes FineVLA a flagship contribution to the field.

Xintong Hu & Xuhong Huang Equal-contribution first authors; XLANG Lab, University of Hong Kong. Led the technical execution of the full FineVLA framework — data pipeline, benchmark construction, policy training, and real-world evaluation. Both are positioned as rising researchers at the intersection of VLMs and robot policy learning.

Shuai Bai Qwen Team, Alibaba Inc. Co-author representing Alibaba's involvement; Bai is also a lead author on the Qwen3-VL technical report (cited as Bai et al., 2025 in Section 6). His involvement signals that Alibaba's Qwen vision-language models are being actively developed toward robotics applications, not just general video understanding.

5. Operating Insights

Retrofit Your Existing Demo Data Before Collecting More

The highest-ROI action from this paper is applying the FineVLA-Tool pipeline to data you already have. The paper shows that 47,159 trajectories selected and re-annotated from 972,247 raw demos outperform the full raw dataset in policy training. "This reduces 972,247 raw trajectories to 47,159 representative samples while preserving diversity in manipulation strategies, object interactions, and motion patterns" (Section 2.1). For teams with large internal demo corpora, running DTW-based clustering plus fine-grained annotation on a representative subset is likely more valuable than collecting 10× more raw demonstrations. The open FineVLA-Tool pipeline makes this immediately feasible.

Build Your Real-World Eval Suite Around Execution Factors, Not Just Task Success

Standard robot evaluation (did the task complete?) hides catastrophic failures in instruction compliance. A robot that completes "pick up the object" using the wrong arm, wrong approach angle, or wrong grasp point is not deployable in constrained industrial or consumer environments. The paper's real-world steerability benchmark — "paired task variants share nearly the same visual scene but require different behaviors according to the instruction, such as choosing a different object color, object pose, approach direction, rotation direction, or active arm" (Section 4.1) — is a template any team can adapt. Before claiming a system is "instruction-following," measure compliance on each execution factor independently.

Use the 1:1 FG-to-Raw Mixing Ratio as Your Starting Point

For teams already building VLA training pipelines, the paper provides a concrete hyperparameter recommendation backed by systematic ablation across three architecture/dataset configurations. Start at FG:Raw = 1:1 and tune toward 1:2 based on your data scale. "The best setting, FG:Raw = 1:1, reaches 86.8%/82.5% on AlohaMix-OFT Easy/Hard, a gain of +15.0/+11.1 over the Raw-only baseline" (Section 4.3). This is not a one-off result — it held across RDT-OFT, RDT-GR00T, and AlohaMix-OFT configurations.

6. Overlooked Insights

The Gripper Weight Signal Is Doing Heavy Lifting in Data Clustering

Buried in the appendix is a methodological detail with outsized implications for anyone building data pipelines: the DTW clustering that selects representative trajectories uses a gripper weight of 100× the positional weight. "Default weights are w_pos = 1.0, w_rot = 1.0, w_grip = 100.0. The high gripper weight ensures that gripper open/close transitions—which are critical for distinguishing manipulation strategies—are not overwhelmed by continuous motion differences" (Appendix A.1.4). This means the clustering is fundamentally organizing demonstrations by contact strategy rather than spatial path. Teams building deduplication pipelines for robot data should adopt this weighting scheme — naive trajectory similarity metrics that treat position and gripper state equally will cluster together demonstrations with fundamentally different manipulation strategies.

Compositional Generalization Remains Completely Unsolved

The paper's OOD probe — asking the robot to use its left arm to place into the right bowl, an actor-target binding not seen in training — scored 0/100 under raw supervision and only 10/100 under the best fine-grained model. "The FG:Raw = 1:2 and 1:1 settings achieve the highest OOD scores (5 and 10 respectively), compared with 0 for Raw-only, suggesting that mixed supervision strengthens individual factor grounding. However, this does not translate into full task completion because the policy still fails to bind the selected arm to the unseen target receptacle" (Section 5.4). This is a hard ceiling for deploying steerable VLA systems in real environments where novel instruction combinations are common. The gap between "following known instruction patterns" and "composing novel instruction factors" is not closed by this work — and any investor or builder claiming their system handles open-ended human instruction should treat this as a red flag for current capability claims.