RobotMover: Learning to Move Large Objects From Human Demonstrations

1. Key Themes

Cross-Embodiment Imitation Learning via Interaction Chain

The paper introduces the "Interaction Chain," a low-dimensional, morphology-agnostic representation that captures how force is transmitted from an agent's core to an object. This allows a quadruped robot (Spot) to learn manipulation skills from human motion capture data, bypassing the need for robot-specific demonstrations. As stated in the paper: "By using the Interaction Chain as the imitation reference, we eliminate the need for manually defining whole-body correspondences between the human and the robot, enabling more scalable and transferable policy learning." (Section I)

Sim-to-Real Transfer Without Visual Feedback

The control policy operates at 20Hz using only proprioceptive data (joint angles, velocities, gripper contact) and the object's target velocity, deliberately excluding visual input. This design choice was made to improve the reliability of transferring policies from simulation to the real world. The authors note: "The observation space excludes direct object state information—such as the object’s orientation, velocity, or visual input—thereby facilitating sim-to-real transfer, improving generalization across object variations, and enabling deployment without requiring external object sensing." (Section V-C)

Outperforming Teleoperation and Learning Baselines

RobotMover outperforms both learning-based methods (RL, RL-EE, RL-IK) and teleoperation in real-world hardware tests. It achieved a maximum linear velocity of 0.75 m/s and angular velocity of 0.5 rad/s, while baselines struggled with stability and collisions. The authors state: "RobotMover outperforms all baselines over all metrics... RobotMover policy is the only policy that able to move a large table and a heavy rack with success rate over 50%." (Section VI-D)

2. Contrarian Perspectives

Visual Feedback is a Hindrance for Dynamic Manipulation Sim-to-Real Transfer

While the industry trend pushes heavily toward vision-language-action models and complex visual perception stacks, this paper argues that for dynamic, contact-heavy manipulation, vision can actually hurt performance and sim-to-real transfer. The authors explicitly chose to omit vision because the camera view is often obstructed during manipulation, and relying on it would degrade performance. They state: "our control policy does not incorporate visual feedback. This reliance on manual initialization limits the system’s autonomy... [but] facilitates sim-to-real transfer, improving generalization across object variations" (Section V-C, VII).

Teleoperation is Insufficient for High-Speed Dynamic Control

Many robotics companies rely on teleoperation for direct deployment or data collection. This paper demonstrates that human operators cannot match the reactive control of a learned policy when moving heavy, dynamic objects. The authors explain: "Teleoperation heavily depends on the operator’s skill level... The RobotMover policy controls both the arm and root movements simultaneously while measuring the robot’s state at 20 Hz. This allows it to responsively adjust its actions to compensate for disturbances or oscillations—an ability that is nearly impossible for a human operator to achieve with the same level of precision." (Section VI-D)

3. Companies Identified

Boston Dynamics

Description: Manufacturer of the Spot quadruped robot. Why relevant: The entire hardware validation of the RobotMover framework is conducted on the Spot platform, demonstrating that their learned policies can control Spot's 19 motors for whole-body coordination. "We use the Boston Dynamics Spot robot, a quadrupedal robot equipped with a robotic arm mounted on its body as our platform." (Section V)

Meta (FAIR)

Description: Fundamental AI Research lab at Meta. Why relevant: The research was conducted as an internship at FAIR, and several authors are FAIR researchers, indicating Meta's continued investment in embodied AI and mobile manipulation. "Work done during an internship at FAIR, Meta." (Section I)

Genesis-Embodied-AI

Description: Developers of the Genesis physics engine. Why relevant: The simulation environment used to train the policies via reinforcement learning is built on the Genesis simulator, highlighting its utility for parallel environment training. "Our simulation environment is built on the Genesis simulator [43]." (Section V-B)

4. People Identified

Tian-Yu Li

Lab/Institution: Georgia Institute of Technology / FAIR, Meta Why notable: Lead author of the paper, developed the RobotMover framework during an internship at FAIR. Focuses on cross-embodiment learning and legged robot control.

Sehoon Ha

Lab/Institution: Georgia Institute of Technology Why notable: Co-author and professor at Georgia Tech. His lab focuses on legged robot learning, motion retargeting, and physics-based character control, which are foundational to this work.

Xavi Puig

Lab/Institution: FAIR, Meta Why notable: Co-author and researcher at FAIR. Involved in mobile manipulation and embodied AI research, contributing to the real-world deployment and system integration aspects of the paper.

5. Operating Insights

Leveraging Human Motion Capture Data for Non-Humanoid Robots

By abstracting interactions into the "Interaction Chain," companies can use existing, abundant human motion capture datasets (like OMOMO) to train robots with entirely different morphologies (like quadrupeds). This bypasses the expensive and slow process of collecting robot-specific teleoperation data for heavy objects. "RobotMover formulates the manipulation problem as imitation learning using a simplified spatial representation... to capture essential human–object interaction dynamics in a morphology-agnostic way." (Abstract)

Prioritizing Dynamics Randomization Over Shape Randomization

When training in simulation, randomizing object shape is computationally expensive because it requires recalculating contact states. The authors found that randomizing mass and friction is sufficient to generalize to new object shapes in the real world, saving significant compute time. "We argue that geometric differences are ultimately reflected in dynamic variations once the object is grasped, and these dynamics can be effectively randomized through variations in friction, mass, and initial poses, without requiring explicit shape randomization." (Section V-B)

6. Overlooked Insights

Cross-Object Generalization Without Fine-Tuning

A policy trained exclusively on chairs was successfully used to move a large, heavy trash cart with asymmetrical weight distribution, without any fine-tuning. This suggests that the learned policy captures fundamental object-pushing dynamics rather than memorizing a specific object's shape, a highly valuable trait for real-world deployment. "The policy used in this experiment is the same policy trained for moving chairs, which are relatively lighter than the trash cart... The trained policy adapts to the cart’s dynamics without fine-tuning." (Section VI-E)

The "Grasping" Bottleneck Remains Unsolved

Despite the impressive manipulation results, the system is not fully autonomous. It requires a human to manually position the robot to establish the initial grasp before the policy can take over. This highlights a critical gap in the autonomy stack that must be solved before commercial deployment. "On hardware, the robot is manually positioned to achieve an initial grasp... This reliance on manual initialization limits the system’s autonomy in real-world applications." (Section VII)