Heavy Metal Acrobatics: Learning Dynamic Control for Skid-Steer Loaders
- 01RL + sim-to-real now works on 5-tonne hydraulic machines, not just lab robots
- 02The sim-to-real recipe is cheap, and that is the real story
- 03Hydraulic delay and nonlinearity can be learned rather than modeled
- 04RL beat a competent MPC baseline on the hard parts, and matched it on the easy ones
- 05Modular skills with hand-written switching are enough for now
- 06The same recipe generalized to a very different platform
1. Key Themes
RL + sim-to-real now works on 5-tonne hydraulic machines, not just lab robots
This is the first demonstration that the legged-robot recipe (train in simulation, deploy zero-shot) transfers to a full-size, operational hydraulic machine. The team trained policies for a John Deere 332G and ran them on the real machine to "swing the machine up onto two wheels and, while balancing on two wheels, drive over rough terrain, trace out specified trajectories, spin around, and swing the bucket back and forth before landing it safely back onto four wheels" (Abstract). Prior RL work on heavy equipment covered "statically stable tasks" such as rock-breaking and excavation (Introduction). This one pushes into dynamic, underactuated, contact-switching behavior. If you're underwriting a heavy-equipment autonomy company, the technical risk of "can learning handle hydraulic dynamics?" just dropped.
The sim-to-real recipe is cheap, and that is the real story
The simulator is crude. The rigid-body model is "a cuboid main body with four cylindrical wheels with uniform density," measured with "a tape measure" and "truck scales" (Methods, Rigid-Body Model). Fidelity comes from one learned component: a small actuator network (3×32 fully connected layers) trained on "1.5 h" of random command data, plus 20 minutes of balancing data (Methods, Actuator Model). Policies are 128×128 networks trained on a single workstation with an RTX 2070 Super (Methods, Reinforcement Learning; Table S.6). The runtime computer is "a standard Linux laptop [Intel Core i7 CPU, 16 GB RAM, no GPU]" (Results). The barrier to entry for a new machine is hours of data collection and a desktop GPU, not a multi-year modeling effort.
Hydraulic delay and nonlinearity can be learned rather than modeled
The paper argues first-principles hydraulic modeling is unnecessary: "Hydraulic actuators exhibit strong nonlinearities and delays, which are very laborious to model from first principles and specific to every machine" (Methods, Actuator Model). The learned model uses 1.8 s of command history and predicts wheel speed 0.02 s ahead. Table S.2 reports validation errors around 0.08 rad/s mean absolute error, and a measured command-to-wheel-speed lag of 0.24 s. That lag is the type of problem that breaks classical controllers, and the policy absorbed it.
RL beat a competent MPC baseline on the hard parts, and matched it on the easy ones
Table 2 shows parity on static balancing, spinning and arm movements. RL wins on swing-up, soft landing, non-flat terrain (75 cm bumps, with the controller "only trained on flat terrain"), and higher velocity tracking (±4 vs ±3.5 rad/s). The explanation matters for engineering planning: for contact-rich behaviors, "with the RL approach this can be achieved by simply changing the reward definition and initialization procedure while leaving the overall training procedure unchanged" (Discussion), whereas MPC needs dedicated planners and hand-extended models.
Modular skills with hand-written switching are enough for now
Rather than one monolithic policy, they trained four skills and composed them using "hand-crafted switching conditions," for example switching from swing-up to balance at -30° pitch. It works "because the state distributions are overlapping between skills" (Discussion; Methods, Maneuvers). This is a pragmatic architecture for anyone shipping, since each skill is separately testable and tunable.
The same recipe generalized to a very different platform
A 100 kg electric platform ("Super Mega Bot") was trained to balance on two wheels with a 5.5-minute actuator dataset and deployed without modification (Supplementary Section S.2). That makes this look like a method, not a one-off demo.
2. Contrarian Perspectives
You don't need a good model of the machine to control it dynamically
Conventional wisdom in heavy equipment says high-fidelity hydraulic models are a prerequisite for precise control. The authors start "without any prior machine-specific knowledge or a preexisting model of the machine" (Results) and rely on a learned actuator model plus domain randomization. Their rigid-body randomization is tiny (center of mass ±0.002 m in x and ±0.02 m in z, inertia ±10%; Table S.4), which suggests the actuator model carries most of the load. Companies selling heavy modeling and system-identification effort as a moat should worry.
Engineered MPC is not the safe choice for dynamic tasks
Most industrial teams treat MPC as the dependable option and RL as research. Here MPC failed at swing-up, soft landing and rough terrain, and "the MPC controller without task-specific extensions struggles to compensate for such terrain disturbances" (Discussion). Their MPC was admittedly simple, and the authors note "this simple MPC baseline" couldn't do swing-up or landing; a human operator moved the machine on and off two wheels for the MPC tests (Supplementary Section S.1). Still, the claim that learned controllers are more robust to unmodeled disturbances, because domain randomization bakes that robustness in, cuts against typical assumptions.
Test heavy machines through extreme dynamics, not around them
The usual instinct is to keep large machines in conservative operating envelopes. This paper deliberately works at the actuation limit ("full-speed motion and maximum control inputs," Introduction), arguing that skills learned there are building blocks for practical tasks like "autonomous self-loading onto trucks without ramps." Most operators would call that reckless. The counterpoint is that training in simulation moves the risk off the physical machine, though the hardware still needed a rear wheelie bar and bucket wheels as safety devices (Figure 2).
3. Companies Identified
John Deere
Manufacturer of the 332G skid-steer loader used for all experiments. Relevant because a major OEM's production machine was successfully automated through a retrofit: joystick signals were intercepted and a laptop sent commands (Results). The disclaimer states "Nothing in this paper constitutes an endorsement by Blue River Technologies (BRT) or John Deere to attempt these operations" (Disclaimer). Deere's own platform is evidently amenable to third-party learned control via its low-level interfaces.
Blue River Technology (BRT)
John Deere-owned precision agriculture/ML subsidiary; several authors are affiliated (Affiliation 2, Santa Clara). Signals that Deere's internal ML organization is actively funding RL for heavy-machine control, and provided access to the machine and test infrastructure (Acknowledgments).
Gravis Robotics AG
ETH spin-off in Opfikon, Switzerland; authors Egli and Jelavic are affiliated (Affiliation 3). Commercially relevant as a startup in autonomous heavy machinery connected to the group's prior excavator RL work (refs [13], [15]).
Intel (Intelligent Systems Lab)
Affiliation 4 for Müller, Sener and Koltun. Indicates continued corporate research investment in learned control and sim-to-real methods.
ETH Zürich Robotic Systems Lab (RSL)
Academic home of the lead group (Affiliation 1); source of the legged-locomotion RL lineage this paper extends (refs [6]-[9]).
Inspectorbots
Maker of the Super Mega Bot used in the case study: "an electric, heavy-duty four-wheel drive (4WD) platform ... weighs 100 kg and has a towing capacity of 1.25 tonnes" (Supplementary Section S.2; ref [36]). Relevant as a platform demonstrating the method generalizes to smaller electric vehicles.
Nvidia / AMD
Training hardware: "Nvidia GeForce RTX 2070 Super GPU and an AMD Ryzen 5 2600 CPU" (Methods, Actuator Model). The point is how modest this compute is.
Component suppliers (Aceinna, MicroStrain by HBK, Same Sky)
IMUs and encoders (MTLT305E/D, MV5-AR, AMT10; refs [37]-[39]). Commodity sensing was sufficient, with no exotic perception stack.
Guinness World Records
Source of the human benchmark: balancing, wheelie and spin records (refs [16]-[18]). Useful as a credible, outside yardstick of operator-level skill.
4. People Identified
Marco Hutter
ETH Zürich RSL; senior author. Led the line of work on ANYmal locomotion and RL for hydraulic excavators: "Learning agile and dynamic motor skills for legged robots" (ref [6]), plus excavator automation (refs [13], [15]). The paper explicitly credits that lineage with the actuator-network approach: "As shown by Hwangbo et al. [6], even sophisticated analytical models fail to capture the dynamics of series-elastic drives."
Vladlen Koltun
Intel Intelligent Systems Lab; co-author on the foundational legged RL papers (refs [6]-[9]) and champion-level drone racing (ref [10]). Notable as a through-line connecting agile flight, legged locomotion and now heavy machinery.
Pierre-Jean Lorentz
Lead/corresponding author, ETH RSL and Blue River Technology. Straddles academia and Deere's ML organization, which suggests a path from research to product.
Pascal Egli and Edo Jelavic
ETH RSL / Gravis Robotics. Egli authored the prior RL excavation work (refs [13], [15], [22]); Jelavic authored the multi-contact planners for legged-wheeled systems (refs [19], [35]) that the paper cites when explaining why MPC struggles with contact switching.
Matthias Müller and Ozan Sener
Intel Intelligent Systems Lab co-authors; Müller also co-authored the drone-racing RL work (refs [10], [11]).
Jemin Hwangbo
Cited for the actuator-network approach (ref [6]) and the RaiSim simulator (ref [26]) used here; his contact-solver is the engine the policies trained in.
5. Operating Insights
Invest in data collection for the actuator, not in a better simulator
The recipe is concrete. Spend about 1.5 hours exercising every actuator with step, sine, ramp and zero inputs (Table S.1 gives the mix), add 20 minutes of data from the target behavior, and fit a small MLP. The authors found the balancing data "slightly improved the performance," so closing the loop with task-relevant data is worth the small cost. Note the history window (1.8 s, 30 points) was "chosen empirically," so expect to tune it per machine. For a heavy-equipment startup, this is the per-machine onboarding cost to benchmark against.
Training-range coverage determines reliability, so test outside it
The one real failure mode is instructive. The policy was trained with arm angle randomized between 29° and 31° and "did not observe the arm configuration." Success was 100% at 28° and 31° but 78.6% (11 of 14) at 34°, with the bucket hitting the ground (Table 1). Before deployment, map the physical operating envelope (payload, arm pose, terrain) against the randomization ranges. A policy blind to a state variable is fragile once that variable leaves its training distribution. If your product carries variable payloads, either expose the variable to the policy or randomize widely.
Plan for a modular skill library with a supervisory layer
The authors deliberately trained separate skills "to keep the tuning effort and number of parameters low." The roadmap they sketch (hierarchical RL for skill selection, distilling skills into one policy, adding LiDAR/camera perception) is the natural path to commercial tasks like obstacle negotiation and truck self-loading. Their handoff rules (for example, pitch ≤ -30°) are simple but depend on overlapping state distributions, so make that overlap a design requirement and test the seams, not just each skill.
6. Overlooked Insights
Sensor noise injection was essential, and the sensing suite is tiny
"Controllers trained without sensor noise in simulation did not perform reliably once deployed in the physical world" (Methods, Observations and Actions). Meanwhile, the policy sees only pitch, pitch rate, average wheel speed and previous action (for the swing-up, as histories). There is no vision, no joint-pressure sensing, and no arm angle. This means a cheap IMU and encoder set drives extreme behavior, but it also means the authors' success depends on noise modeling being right; budget engineering time for it.
The machine had permanent safety hardware, and the hard-landing problem is real
Results were obtained with a rear "wheelie bar" and wheels on the bucket to prevent tipping, and these were intentionally excluded from the collision model so the agent couldn't exploit them (Methods, Rigid-Body Model). Also, no-command landing "resulted in a severe impact" with rebound (Results, Soft Landing). Any commercialization path for these dynamic skills needs a safety-certified answer for failure recovery, and the paper's own disclaimer distances Deere from the maneuvers. The practical finding for investors is that the near-term value is likely in the robustness and control-synthesis tooling (the paper's argued applications such as balance-assisted self-loading, rough-terrain handling, and recovery maneuvers are future work), not in acrobatics itself.