Assembling Two Parts in One Hand
1. Key Themes
Single-Handed Assembly as a New Benchmark for Dexterity
The paper introduces a fundamentally new task family for robotic manipulation: in-hand assembly. Instead of using a second arm or an external fixture to hold one part while another is inserted, the system requires a single dexterous hand to simultaneously stabilize one object while manipulating another to mate them. This moves robotic capabilities beyond simple pick-and-place or in-hand rotation, directly mimicking human dexterity. As the authors state: "We introduce in-hand assembly as a challenging dexterous manipulation setting. By requiring one hand to coordinate multiple fingers across two interacting objects, it provides a task family for evaluating coordinative dexterity in robotic manipulation systems." (Section 1)
Role-Based Finger Coordination via RL
The system achieves complex assembly by assigning specific functional roles to different fingers, driven by a unified reinforcement learning framework. The thumb and index finger form a "pinch" to manipulate the held object, while the middle, ring, and pinky fingers form a "grasp" to cage the fixed object and resist reaction forces. The authors note: "The resulting behavioral specialization... is: the held set Fh = {thumb, index} manipulates Oh at its top and provides the fine alignment and goal-pose-reaching motion. The fixed set Ff = {middle, ring, little} cages Of and resists the reaction wrenches generated by Fh..." (Section 3.1)
Zero-Shot Sim-to-Real Transfer with Robustness to Occlusion
The policies are trained entirely in simulation and transfer directly to real-world hardware using only a single depth camera. Because fingers naturally occlude objects during fine manipulation, the system relies on domain randomization and a fusion of historical proprioception (joint angles) with visual state estimation to maintain robustness. The authors report: "Trained purely in simulation, the policies transfer zero-shot to hardware with a single camera, demonstrating robustness to state-estimation errors caused by occlusion." (Abstract)
Hand Morphology Dictates Assembly Success
The paper demonstrates that in-hand assembly is a rigorous test of robotic hand hardware design. It reveals that simply having a high number of degrees of freedom (DoF) is insufficient if the hand lacks the right physical dimensions or joint types. The authors found that "The Sharpa Hand and Wuji Hand perform best, owing to their human-like dimensions, high DoF, and wide joint ranges." (Section 4.2) Conversely, hands lacking a fifth finger or abduction/adduction joints failed at basic alignment and insertion tasks.
2. Contrarian Perspectives
High Degrees of Freedom Do Not Guarantee Dexterity
Many robotics companies chase ever-higher DoF counts as a proxy for capability. This paper challenges that assumption, showing that a hand with 12 DoF (XHand) fails entirely at assembly because it lacks abduction/adduction joints, while a 16-DoF hand (Allegro) fails at deep insertion because it lacks a fifth finger. The authors explicitly state: "The XHand is close to human-hand size but has fewer degrees of freedom—most of its fingers lack abduction/adduction joints—and it fails to insert the objects, and the objects cannot be aligned in the xy-plane in many cases." (Section 4.2) Hardware morphology and joint range matter more than raw DoF.
Open-Loop Replay of Successful Trajectories is Useless for Contact-Rich Tasks
A common shortcut in robotics is to record a successful teleoperated trajectory and replay it to perform a task. This paper shows that for contact-rich, multi-object assembly, open-loop replay is almost entirely useless due to the accumulation of sim-to-real dynamic errors. The authors found: "Open-loop replay achieves successful alignment only three times. Without feedback, the action sequences cannot compensate for sim-to-real dynamics, so early deviations accumulate, and the parts fail to align." (Section 4.5) Closed-loop feedback is strictly necessary.
Multi-Frame Human Demonstrations are Unhelpful for Fine Finger Motion
Learning from human video is a massive trend in Physical AI. However, the authors argue that the embodiment gap between human and robot hands is so severe that trying to track multi-frame human finger motions provides little usable guidance. Instead, they only use a single static snapshot of a human hand pose for initialization. They explain: "For in-hand assembly, however, we apply such references only as single-frame snapshots: the fine, intricate finger motion of in-hand manipulation is susceptible to the embodiment gap and extraction noise, so multi-frame references offer little usable guidance for finger motion." (Section 3.4)
3. Companies Identified
Sharpa Robotics
Maker of the 22-DoF Sharpa Wave Hand used in the main simulation and real-world experiments. Relevant because it proved to be one of the only hands capable of completing the tasks due to its human-like dimensions and wide joint ranges. "We use a 22-degree-of-freedom (DoF) Sharpa Wave Hand [44] in most of the simulation and real-world experiments..." (Section 4.1)
Franka Robotics
Maker of the Franka Research 3 arm used to mount the hand. Relevant as the standard industrial/research arm providing the base for the dexterous hand. "...which is fixed on a Franka Research 3 arm [45]." (Section 4.1)
NVIDIA
Maker of IsaacSim and the PhysX SDK. Relevant because the entire pipeline relies on NVIDIA's simulation infrastructure for contact-rich training, though the authors note PhysX limitations with planar contacts. "The simulation experiments are conducted in IsaacSim [46]." (Section 4.1)
Intel
Maker of the RealSense D435 camera. Relevant because the system achieves zero-shot sim-to-real transfer using only this single stereo depth camera for state estimation. "We rely on a single RealSense D435 camera [47] to perform state estimation." (Section 4.1)
Wuji Tech
Maker of the 20-DoF Wuji Hand. Relevant as a competitive hand that also performed well on the benchmark. "Beyond the Sharpa Hand used in our main experiments, we evaluate the Wuji Hand [48]..." (Section 4.2)
Wonik Robotics
Maker of the 16-DoF Allegro Hand. Relevant because its evaluation revealed a critical hardware limitation: lacking a fifth finger prevents deep z-axis insertion. "...the Allegro Hand [49]..." (Section 4.2)
RobotEra
Maker of the 12-DoF XHand. Relevant because its evaluation showed that lacking abduction/adduction joints causes failure in xy-plane alignment, highlighting a design flaw for complex tasks. "...and the XHand [50]." (Section 4.2)
4. People Identified
Liuao Pei & Tianyue Wu
Equal-contribution first authors from The University of Hong Kong and The Hong Kong University of Science and Technology (Guangzhou). They led the system design, simulation, and real-world experiments for this dexterous manipulation pipeline.
Ping Luo
Corresponding author and Professor at The University of Hong Kong. Notable for his extensive work in deep learning and computer vision, bringing expertise in visual state estimation and policy representation.
Jie Song
Corresponding author and Professor at HKUST(GZ) and HKUST. Notable for his lab's focus on robotic manipulation, grasping, and human motion analysis, bridging the gap between human dexterity and robotic hardware.
Hui Zhang
Co-author from ETH Zurich. Brings expertise in robotic grasping and manipulation, contributing to the geometric retargeting and reward design aspects of the system.
5. Operating Insights
Hardware Selection Requires Task-Specific Morphological Benchmarking
When selecting a robotic hand for assembly or complex manipulation, do not rely on DoF count as a primary metric. Ensure the hand has 5 fingers, human-like dimensions, and critically, abduction/adduction joints. A hand missing a pinky will fail at deep insertion, and a hand missing abduction/adduction will fail at lateral alignment. As the authors note: "The Sharpa Hand and Wuji Hand perform best, owing to their human-like dimensions, high DoF, and wide joint ranges." (Section 4.2)
Robustness to Occlusion Requires Fusing Proprioception with Vision
Visual state estimation will fail during contact-rich, multi-finger tasks due to occlusion. Policies must be trained with domain randomization and recurrent architectures (like LSTMs) to implicitly calibrate noisy visual estimates using proprioceptive history. The authors found that performance remained above 60% even under strong noise (3 cm standard deviation), likely because "this robustness may stem from the policy’s maintenance of proprioceptive history." (Section 4.4)
Soft Finger Pads Create a Sim-to-Real Gap in Pinch Grasps
If your application relies on pinch grasps, be aware that current rigid-body simulators cannot model planar (patch) contacts well; they degenerate pinch grasps into unstable point contacts. While soft finger pads alleviate this in the real world, they introduce a sim-to-real gap that prevents the policy from learning decisive pinch behaviors in simulation. "Soft finger pads alleviate the instability in the real world, but introduce a sim-to-real gap. Consequently, the policy cannot learn decisive pinch behaviors in these configurations during simulation, leading to failures in real-world deployment." (Section 4.5)
6. Overlooked Insights
Manual Object Placement is Still Required for Initialization
Despite the advanced RL policy, the system is not fully autonomous. It currently requires a human to manually place the objects in the hand's fingers before the neural network control is activated. This limits immediate deployment in fully automated pipelines. The authors state: "In our experiments, we first fixed the wrist of the robotic hand... then manually placed Of between the middle, ring, and pinky fingers, and placed Oh between the index finger and the thumb." (Appendix B)
Joint Overheating is a Real Risk in Contact-Rich Multi-Object Manipulation
Manipulating two objects simultaneously causes jamming between parts that can rapidly overheat and damage robotic hand hardware. The system requires a tracking-error-driven stall-relax loop to monitor commanded vs. measured joint positions and slightly relax targets to keep motor currents below thermal limits. "Otherwise, jamming between objects caused by manipulating two objects simultaneously would lead to rapid overheating of the robotic hand and damage to the hardware." (Appendix B)