SeededGrasp: Language… | arXiv Physical AI Research Summary

1. Key Themes

Decoupling Semantic Reasoning from Geometric Execution via Seed Points

The core architectural innovation is using a VLM to predict a single 3D "seed point" — a pixel coordinate projected onto the scene point cloud indicating where the grasp should be centered — which then conditions a lightweight flow-matching model for grasp pose generation. This avoids the two dominant paradigms: (a) having the VLM directly predict grasps (which lacks 3D spatial awareness) or (b) end-to-end training of the VLM with the grasp model (which requires expensive language-annotated datasets and massive compute). The paper states: "Our architecture decouples high-level semantic reasoning from low-level geometric execution, enabling multi-embodiment support while bypassing the need for expensive end-to-end training" (Abstract). The result: 72% success in simulation and 78% in real-world experiments, with the VLM used zero-shot without fine-tuning or few-shot examples (Section 4.3).

Multi-Embodiment Grasping from a Single Model

A single trained model generates grasps for three morphologically distinct grippers — the Franka Panda (parallel-jaw), Robotiq 3-Finger (underactuated), and Allegro Hand (16-DOF dexterous) — by conditioning on gripper point cloud geometry and a learnable per-gripper query vector. The model achieves 71.38%, 72.16%, and 72.07% success rates respectively (Table 2, Right), demonstrating that performance is consistent across embodiments rather than degrading for more complex hands. The ablation in Table 4 shows that removing either the robot point cloud encoding or the query vector significantly hurts performance for the more kinematically complex Robotiq and Allegro grippers (dropping to 59-67%), while the Franka Panda remains largely unaffected — suggesting that simpler grippers are robust to representation choices but dexterous hands require richer encoding.

First Multi-Embodiment Cluttered Scene Grasping Dataset

The paper releases a dataset of 2.56M grasps across 610 cluttered scenes, 334 objects, and 3 grippers (Table 1). This fills a gap the authors identify explicitly: "none of the existing datasets provide data for multi-embodiment grasping in cluttered scenes" (Section 2, Datasets subsection). The dataset was built by transforming mid-air grasps from the MultiGripperGrasp dataset into cluttered tabletop scenes using heuristic collision checks (approach angle ≥0.5 rad, table clearance ≥0.5 cm, interpenetration threshold of 10 mm). The dataset is notably skewed toward the Franka Panda (1.61M training grasps vs. 246K for Robotiq and 161K for Allegro), but the authors found this imbalance did not require correction: "further generation was found to be unnecessary for the scope of this work" (Section 3).

VLM Choice is a Critical Engineering Decision, Not a Commodity

The paper ablates three VLMs for seed point prediction and finds dramatic differences: Gemini 3.1 Flash achieves 71.87% average success, Qwen 3.5 Flash achieves 54.42%, and GPT 5.4 achieves only 42.61% (Table 5). This is a ~30 percentage point spread — meaning the choice of VLM can make or break deployment. Notably, "using a higher reasoning level did not help performance" (Section 5.2, VLM ablation), suggesting that raw spatial grounding capability, not chain-of-thought reasoning, is what matters for this task.

2. Contrarian Perspectives

End-to-End VLA Training Is Not Always the Answer

The robotics industry has been heavily investing in end-to-end vision-language-action (VLA) models — π0, RT-2, GraspVLA — that train the entire perception-to-action pipeline jointly. This paper argues the opposite: keeping the VLM frozen and using a simple seed point as the interface is more data-efficient, requires no language-annotated training data, and scales naturally as VLMs improve. The authors state: "Our method bridges both approaches by introducing a seed point as an efficient interface connecting a pretrained VLM with a learned flow matching grasp generation module, eliminating the need for a language-annotated dataset" (Section 2, Language-grounded grasping). The modular design also means VLM improvements automatically benefit the system: "the approach is not specific to the selected VLM in this modular design, enabling natural performance scaling from future VLM improvements" (Section 4). For a startup, this means you don't need to retrain a massive VLA model every time a better foundation model drops — you swap in the new VLM and keep your grasp model.

Zero-Shot VLM Spatial Grounding Beats Trained Seed Point Modules

The paper finds that an off-the-shelf VLM predicting seed points zero-shot outperforms a purpose-trained seed point generation module (DGN2.0's Graspness Score Module). Specifically, in Table 2 (Left), DGN2.0 with VLM seeds achieves 66.67% success vs. 53.15% with its own trained Graspness module — a 13.5 percentage point improvement. SeededGrasp with VLM seeds reaches 72.97%. This challenges the assumption that task-specific trained modules are always better than general-purpose foundation models. The implication: for companies deciding whether to invest in training custom perception modules, a well-prompted VLM may deliver better results at a fraction of the cost.

A Single Multi-Embodiment Model Is More Data-Efficient Than Specialized Models

Most robotics companies deploy a separate grasping model per end-effector. This paper provides evidence that a shared model across embodiments is not just simpler to maintain but actually more data-efficient: "Supporting multiple embodiments offers the additional advantage of improving grasping performance, making a single multi-embodiment model more useful and data-efficient than specialized ones" (Section 1, citing [8, 9, 10, 11]). The dataset ablation (Section 5.2, Figure 10) shows that reducing training data disproportionately hurts the Robotiq and Allegro grippers, while the Franka Panda is robust — suggesting that cross-embodiment training provides a regularization benefit where data-scarce grippers leverage shared representations learned from data-rich ones.

3. Companies Identified

Google DeepMind, AI research lab — Maria Attarian (co-author) is affiliated with Google DeepMind in addition to UofT/Vector. Relevant because the paper uses Gemini 3.1 Flash as its VLM, which is a Google product, and the connection suggests potential insider knowledge of upcoming VLM capabilities. The paper does not explicitly discuss Google DeepMind's role beyond the affiliation.

Google (Gemini), VLM provider — Gemini 3.1 Flash is used for all seed point predictions and significantly outperforms alternatives (71.87% vs. 54.42% for Qwen and 42.61% for GPT). The paper states: "Gemini 3.1 Flash outperforms competing architectures" (Section 5.2, VLM ablation). This suggests Google's VLMs currently have superior spatial grounding for robotics tasks.

OpenAI (GPT), VLM provider — GPT 5.4 was evaluated for seed point prediction and achieved only 42.61% average success, the worst of the three VLMs tested. Relevant as a competitive data point for robotics teams selecting VLM backbones.

Alibaba (Qwen), VLM provider — Qwen 3.5 Flash achieved 54.42% average success, placing it between GPT and Gemini. Relevant for teams considering open-weight VLM alternatives.

Franka Emika, robot manufacturer — The Franka Panda parallel-jaw gripper is one of three grippers in the dataset and evaluation. It has the highest volume of viable grasps (1.61M training) due to simpler kinematics: "Its simpler kinematics allow for a higher percentage of grasp poses to pass filtering" (Section 3, Table 1).

Robotiq, gripper manufacturer — The Robotiq 3-Finger underactuated gripper is one of the three training embodiments. It had 246K viable in-scene training grasps, significantly fewer than Franka due to more complex kinematics causing more collisions in cluttered scenes.

Wonik Robotics (Allegro), dexterous hand manufacturer — The Allegro Hand (16-DOF) is the most complex gripper in the study, with 161K viable training grasps. Despite having the least training data, it achieves 72.07% success, demonstrating the model's ability to handle dexterous hands.

Delto, gripper manufacturer — The Delto DG-3F-B three-finger gripper was used in real-world experiments and was NOT in the training data. The authors mapped Robotiq 3-Finger predictions to it "using constant offsets on certain joints and fixed values for the additional joints" (Section 5.3), achieving 78% success. This is a significant finding about cross-hardware transferability.

NVIDIA, simulation platform — Isaac Sim was used for all simulation evaluations (Appendix A.1), with a pass/fail metric based on object lift success (30 cm above table without dropping).

Stereolabs (ZED), camera manufacturer — Two stationary ZED stereo cameras were used to generate point clouds for real-world experiments (Section 5.3).

4. People Identified

Yang Xu, University of Toronto / Vector Institute — Shared first author. Part of the Intelligent Scientific Computing (ISC) lab at UofT. The project page is hosted under the UofT-ISC GitHub organization.

Gurpreet Singh Mukker, University of Toronto — Shared first author. Working on grasping and manipulation research.

Raymond Wang, University of British Columbia — Shared first author. Notable for being at a different institution (UBC) than the primary lab (UofT), suggesting cross-institutional collaboration.

Jasper Gerigk, University of Toronto / Vector Institute — Co-author involved in the work, affiliated with both UofT and the Vector Institute.

Maria Attarian, University of Toronto / Vector Institute / Google DeepMind — Co-author with a dual academic-industry affiliation. Notable because she is also a co-author on GeoMatch (reference [10]), the multi-embodiment grasping method that SeededGrasp builds upon and outperforms. Her presence at Google DeepMind suggests this line of research has industry relevance. The paper cites GeoMatch as a baseline and improves upon it by ~35% in cluttered scenes (Section 5.1.2).

Igor Gilitschenski, University of Toronto / Vector Institute — Senior/corresponding author. His lab (UofT-ISC) produced this work and hosts the project page. Also co-authored GeoMatch [10] and GeoMatch++ [30], establishing a sustained research program in multi-embodiment grasping that this paper extends.

5. Operating Insights

The Seed Point Pattern Is a Practical Architecture for Production Robotics Teams

For CTOs building manipulation systems, the seed point interface — where a VLM outputs a single 3D coordinate that conditions a lightweight generative model — is a deployable pattern today. It requires no VLM fine-tuning, no language-annotated datasets, and the grasp generation model trains in 48 hours on 2 A100 GPUs (Section 5). The VLM can be swapped without retraining the grasp model. The prompt is straightforward (Appendix A.6): a system prompt asking the VLM to output a JSON with normalized pixel coordinates, plus a bird's-eye view image. This means a small team can build a language-conditioned grasping system without the data pipeline or compute budget of a VLA model. The tradeoff: the system currently predicts grasps without arm kinematics awareness, so a separate motion planning layer is needed (Section 6).

Cross-Hardware Transfer Works with Simple Joint Mappings — But Test Before Committing

The real-world experiment used a Delto DG-3F-B gripper that was never in the training data, yet achieved 78% success by mapping Robotiq 3-Finger predictions with "constant offsets on certain joints and fixed values for the additional joints" (Section 5.3). For operators: if your target gripper is morphologically similar to one in the training set, you may not need to retrain. However, the authors note the Delto's fingers are "not as long or thick as the Robotiq ones" (Section 5.3), and the primary failure mode was the gripper being too far from the object. This suggests the mapping approach works but has physical limits — the closer the hardware match, the better.

VLM Selection Should Be Treated as a First-Class Engineering Decision

The 30 percentage point spread between Gemini (71.87%) and GPT 5.4 (42.61%) for seed point prediction (Table 5) means that VLM choice has a larger impact on grasping performance than many architectural decisions. Teams should benchmark multiple VLMs on their specific task before committing. Notably, higher reasoning capability did not help — "using a higher reasoning level did not help performance" (Section 5.2) — so the most expensive or most "intelligent" VLM is not necessarily the best for spatial grounding tasks. The per-gripper breakdown (Table 6) shows the ranking is consistent across all three grippers, suggesting the advantage is systematic rather than task-specific.

6. Overlooked Insights

Performance Saturates at Current Dataset Scale — More Data Won't Help Much

The dataset size ablation (Section 5.2, Figure 10) reveals that grasping success saturates at approximately 71.5% with the full dataset, with "only marginal improvements if the dataset size was further increased." This is a critical finding for resource allocation: if you're building a similar system, doubling your data generation budget will not double your performance. However, the saturation is gripper-dependent — the Franka Panda shows almost no degradation with 25% of data, while Robotiq and Allegro benefit from the full dataset. This suggests that for dexterous hands, data diversity matters more than raw volume, and the bottleneck may be in scene/object variety rather than grasp count.

The Training Data Has a Power Grasp Bias That Limits Real-World Utility on Flat Objects

Buried in the Limitations section: "the current over-representation of power grasps in the MGG dataset limits the model's effectiveness when handling flat objects that sit flush against a surface" (Section 6). This is a significant deployment risk — many real-world warehouse and household objects (books, tablets, flat packaging, trays) are precisely the kind of flat objects that this bias would cause failures on. The real-world results support this: the lowest success rates were for the green bottle (6/10) and large rubber duck (6/10), both of which present challenging geometries for power grasps. Companies evaluating this approach should test specifically on flat, thin, or flush objects before deployment.