ResSafe: Learning Safety Filtering with Residual Reinforcement Learning for Humanoids

1. Key Themes

Decoupling Performance and Safety via Residual RL

The paper's core contribution is splitting humanoid control into two policies: a nominal policy optimized purely for task performance (motion tracking), and a residual policy that learns safety corrections on top. The final action is the nominal action plus a learned residual. This avoids the fundamental problem of single-policy RL where safety and performance rewards compete, leading to suboptimal Pareto trade-offs. As stated in the Introduction: "improving robustness or safety, e.g., through increased domain randomization or stronger penalties for failures, inevitably comes at the expense of degraded performance" (Section 1).

Superior Performance–Safety–Robustness Trade-off

ResSafe empirically demonstrates a better trade-off than single-policy approaches. In simulation, the baseline policy trained without payload randomization achieves 24.95mm tracking error but a 7.29% fall rate under payload. The baseline trained with payload randomization reduces fall rate to 1.20% but degrades tracking to 43.28mm. ResSafe achieves 37.66mm tracking error with only 1.11% fall rate (Table 1). For the safety trade-off, adding safety cost weights to a single policy either doesn't help (fall rate goes from 0% to 1.63% at weight 100) or destroys performance (353.55mm error). ResSafe achieves 68.65mm error with 0.5% fall rate (Table 2).

Generalization Across Unseen Reference Policies

The residual safety policy, when trained across multiple reference policy checkpoints, generalizes to unseen reference policies without retraining. Figure 4(b) shows that multi-checkpoint training maintains robustness when deployed with unseen reference policies, whereas single-checkpoint training causes fall rates to "increase sharply" (Section 5.2). This suggests the safety layer captures general balance priors rather than overfitting to a specific controller.

Validated on Real Hardware Under Payload Disturburbances

The framework is deployed on a Unitree G1 humanoid in the real world across multiple balance motions with and without payloads attached to limbs. In the most challenging case (motion #6337 with payload), the baseline falls 10/10 times, while ResSafe achieves 0/10 falls with 70.01mm tracking error (Table 3). The discriminating hyperplane baseline falls 9/10 times on the same motion.

Implicit Safety Filtering Without Online Optimization

The paper connects residual RL to classical control-theoretic safety filters (CBFs, HJ reachability) but eliminates the need for online optimization or explicit dynamics models. The residual policy "behaves similarly to a min-norm safety filter correction, providing an implicit safety filtering mechanism without online optimization" (Section 1, contributions). This is operationally significant: classical safety filters require solving optimization problems at runtime, which doesn't scale to high-dimensional humanoids (n=58, m=29 for G1).

---

2. Contrarian Perspectives

Single-Policy RL with Reward Engineering Is Fundamentally Suboptimal

Most robotics companies train a single end-to-end policy with dozens of reward terms and tune weights until behavior looks acceptable. This paper argues that approach is structurally flawed because gradient-based RL "typically converge to locally optimal solutions, often resulting in a suboptimal Pareto frontier" (Section 1). The evidence is direct: in Table 2, increasing safety cost weight from 50 to 100 increases fall rate from 0% to 1.63% while destroying tracking performance (353.55mm error). More safety penalty can literally make the robot less safe. The decoupled approach sidesteps this by training safety as a separate optimization.

Domain Randomization for Robustness Hurts More Than It Helps for Tracking

The conventional wisdom in sim-to-real is to add heavy domain randomization to your single policy. This paper shows that approach creates over-conservatism: Baseline* (trained with payload randomization) achieves worse tracking error without payload (44.40mm vs. 24.95mm for the non-randomized baseline) due to "over-conservatism from domain randomization" (Section 5.3, Table 1). ResSafe gets the robustness benefits of randomization without the tracking degradation by isolating safety learning from performance learning.

Learned Safety Filters Can Work Without Hard Guarantees

Classical safe control researchers would argue that safety without formal guarantees (CBF certificates, HJ value functions) is not real safety. This paper explicitly acknowledges it "does not provide hard safety guarantees in the sense of classical safety filters" (Section 7, Limitations) but demonstrates empirically that a learned residual policy still dramatically reduces fall rates in both simulation and real hardware. For commercial deployment, empirical safety improvement from 10/10 falls to 0/10 falls matters more than theoretical certificates.

---

3. Companies Identified

Unitree

• Description: Manufacturer of the G1 humanoid robot (29-DOF, 58-dimensional state)
• Why relevant: The G1 is the physical platform for all experiments. The paper validates that ResSafe works on a commercially available humanoid, not just a custom research platform. The state/input dimensions (n=58, m=29) are specific to G1's configuration.
• Quote: "For Unitree G1, the state and input dimensions are n = 58, m = 29" (Section 4.2)

NVIDIA

• Description: Provider of IsaacGym physics simulator
• Why relevant: IsaacGym is the simulation environment used for all training. This is the standard GPU-accelerated simulator for humanoid RL, reinforcing NVIDIA's position in the Physical AI stack.
• Quote: "Simulation experiments are conducted using the IsaacGym [54] simulator" (Section 5.1)

VESSL AI

• Description: AI infrastructure company providing GPU compute resources
• Why relevant: Supported the research with compute, suggesting the training pipeline requires significant GPU resources. Relevant for operators evaluating infrastructure needs for humanoid policy training.
• Quote: "VESSL AI for supporting this research with GPU compute resources" (Acknowledgments)

---

4. People Identified

Koushil Sreenath

• Lab/Institution: UC Berkeley, Hybrid Robotics Group
• Why notable: One of the most influential researchers in RL-based humanoid and legged robot control. His prior work on versatile bipedal locomotion (referenced as [1]) and the HuB extreme balance framework (referenced as [2]) are foundational to this paper. The reference policy in ResSafe is built directly on HuB.
• Quote: Co-author and equal advisor (†). The HuB pipeline [2] is the basis for the reference policy training.

Claire Tomlin

• Lab/Institution: UC Berkeley, CyPhy Lab
• Why notable: Pioneer in Hamilton-Jacobi reachability and safety-critical control. Her theoretical foundations underpin the safety filter formulation in this paper. Brings control-theoretic rigor to the RL approach.
• Quote: Co-author and equal advisor (†). HJ reachability analysis [8, 9] is the theoretical basis for the safety value functions.

Jason Jangho Choi

• Lab/Institution: UCLA
• Why notable: Equal advising author with expertise in safety filters for black-box systems. His prior work on learning discriminating hyperplanes [6] is the baseline that ResSafe outperforms, showing intellectual honesty in benchmarking against his own prior method.
• Quote: Equal advisor (†). The discriminating hyperplane method [6] is used as a baseline comparison.

Gechen Qu

• Lab/Institution: UC Berkeley
• Why notable: Lead author and corresponding contact. Emerging researcher in the intersection of safe control theory and reinforcement learning for humanoids.
• Quote: Correspondence: qugch@berkeley.edu

---

5. Operating Insights

Architect Your Stack for Decoupled Safety, Not Monolithic Policies

If you are training humanoid or legged robot policies, consider separating your performance policy from your safety policy rather than engineering increasingly complex reward functions. The evidence is clear: single-policy approaches with safety cost weights either don't improve safety or actively harm it (Table 2: weight 100 gives 1.63% fall rate vs. 0% at weight 50, with 6x worse tracking). A residual safety layer trained separately can be updated, fine-tuned, or replaced without retraining your entire performance stack. This has direct implications for iteration speed and deployment safety.

Train Safety Policies Across Diverse Reference Checkpoints for Generalization

The paper reveals a critical training detail: if you train your residual safety policy on a single reference policy checkpoint, it fails catastrophically on unseen reference policies (Figure 4(a)). The fix is training across multiple checkpoints from a diverse motion dataset (10,000 balance motions). For operators, this means your safety layer training pipeline needs to expose the policy to a wide distribution of nominal behaviors, not just the one you plan to deploy. The paper generates diverse motions using the method in [39] and trains on "multiple reference policy checkpoints, which allows it to generalize against unseen unfiltered actions" (Section 4.2).

Payload Randomization Is Essential for Safety Policy Training

During residual policy training, the authors introduce payloads of 0-3kg at wrists, elbows, and shoulders, plus a curriculum of random pushes up to 0.5 m/s. Without this excitation, "the policy tends to learn a very conservative safety filter which is impractical" (Section 4.2). For deployment scenarios where robots carry tools or objects, the safety policy must be trained explicitly against these disturbances. The real-world results confirm: motions with payload (#6337, #8407) show the largest gap between baseline (10/10 and 5/10 falls) and ResSafe (0/10 and 0/10 falls).

---

6. Overlooked Insights

The Safety Policy Is "Blind" — No Visual Perception

The paper acknowledges in Limitations that "the current framework operates as a blind safety filter without exteroceptive perception" and that "human's balancing and safety behaviors rely heavily on visual feedback and environmental awareness" (Section 7). This is a significant gap: the safety policy only uses proprioceptive states (joint positions, velocities, base angular velocity, projected gravity — 122 dimensions total per Table 6). For real-world deployment in cluttered environments, on stairs, or near edges, a proprioception-only safety filter cannot detect environmental hazards. Companies deploying this approach would need to add perception or accept that the safety layer only handles internal balance, not environmental awareness.

The Residual Policy Is Deliberately More Conservative Than Optimal

The residual RL policy exhibits "slight conservativeness" compared to the theoretical min-norm safety filter (Section 4.1, Figure 3(c)). The authors note this "acts as a robustness margin" but also flag it as an open theoretical question (Section 7, Limitation 2). This means the safety layer will sometimes intervene when it doesn't strictly need to, potentially causing jerky or suboptimal motion in edge cases. Operators should expect some performance degradation in scenarios near the safety boundary that isn't captured by the aggregate tracking error metrics.