ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models

1. Key Themes

Continual Off-Policy RL for Robot Foundation Models

The paper introduces ACCRUE, a framework that allows robots to learn new tasks sequentially without forgetting old ones. Instead of resetting the reinforcement learning (RL) system for every new task, ACCRUE carries over lightweight "actor" and "critic" networks. This means the robot can reuse the behavioral and value knowledge it gained from previous tasks, making it faster and more efficient at learning new things. As the paper states, "We propose ACCRUE, a continual off-policy RL framework that carries lightweight actor and critic networks across tasks, allowing each stage to reuse behavioral and value knowledge acquired through prior RL." (Abstract)

Drastic Reduction in Real-World Training Time

For companies deploying robots, the time a robot spends learning a new task on the factory floor is expensive downtime. ACCRUE significantly cuts down this real-world training time by reusing prior knowledge. The authors note, "On two multi-stage real-robot sequences, ACCRUE reduces robot RL interaction time for subsequent tasks by 31.0%–68.5% compared with training a new actor–critic for each task, while maintaining approximately 97% average success on prior tasks." (Abstract) This means a robot can learn a new assembly step in minutes rather than hours, all while retaining its ability to perform older tasks.

Asymmetric Update for Forgetting Mitigation

A major challenge in continual learning is "catastrophic forgetting"—learning a new task overwrites the knowledge needed for old tasks. ACCRUE solves this with an asymmetric update strategy: it only updates the critic (the value estimator) on the current task, and preserves old behaviors by forcing the actor to mimic its past outputs on a small set of retained old data. The paper explains, "We deliberately exclude prior-task samples from critic updates because the small number of retained episodes may not provide sufficient coverage for reliable value estimation on all prior tasks... Instead, ACCRUE preserves prior behavior by distilling the actor toward cached outputs of the preceding-stage actor on the last N episodes retained from each completed task" (Sec 1).

2. Contrarian Perspectives

More Replay Data Isn't Always Better

A common assumption in machine learning is that feeding the model more historical data will improve retention. ACCRUE's ablation studies challenge this, finding that the relationship between retained episodes and performance is non-monotonic. Increasing the replay buffer from 100 to 200 episodes actually degraded performance. The authors conclude, "This non-monotonic trend shows that effective continual post-training does not follow directly from replaying more prior data." (Sec 5.4) This suggests that smart curation of a small memory buffer is more valuable than simply storing massive amounts of historical data.

On-Policy RL is Less Efficient for Continual Learning

Many recent approaches to adapting Vision-Language-Action (VLA) models across sequential tasks rely on on-policy RL, which requires the robot to constantly generate fresh rollouts for every update. ACCRUE argues that off-policy learning—which can reuse previously collected data—is fundamentally more interaction-efficient for continual learning. The paper states, "Recent methods use RL to adapt VLAs across sequential tasks and mitigate forgetting... but provide limited forward transfer. Whereas on-policy updates require fresh current-policy rollouts, off-policy learning can reuse previously collected transitions to improve interaction efficiency." (Sec 1) For real-world deployment, where robot interaction is the bottleneck, off-policy is the superior choice.

3. Companies Identified

NVIDIA

Description: AI computing and robotics hardware/software company. Why relevant: The paper is authored by NVIDIA researchers and heavily utilizes NVIDIA's ecosystem. The simulation experiments use GR00T N1.7 as the base robot foundation model. "1NVIDIA... We use GR00T N1.7 (26) for πbase" (Sec 5.2). This indicates NVIDIA is actively building the tooling and models for continual robot learning.

Physical Intelligence

Description: Robotics foundation model startup (creators of π0 and π0.5). Why relevant: The paper uses Physical Intelligence's π0.5 model as the base policy for real-world hardware experiments on the DROID platform. "For the base policy πbase, we utilize π0.5 (3) rather than GR00T N1.7 owing to its stronger performance on DROID." (Sec 5.3) This highlights π0.5 as a leading base model for real-world manipulation.

Dyna Robotics

Description: Robotics company focused on world-action models. Why relevant: Referenced in the paper's literature review for their work on world-action models (Dyna-2). "Dyna-2: A 1-million-hour scaling law for world-action models" (References). This situates ACCRUE within the broader landscape of both VLA and WAM approaches.

4. People Identified

Changyeon Kim

Lab/Institution: NVIDIA and KAIST Why notable: Lead author of the paper. His work focuses on scaling offline and online RL for robot manipulation, as evidenced by his prior work (DEAS) cited in the paper.

Yashraj Narang

Lab/Institution: NVIDIA Why notable: Co-author and a key researcher at NVIDIA involved in creating the FORGE and AutoMate benchmarks used in the paper. He is a central figure in NVIDIA's manipulation research.

Kimin Lee

Lab/Institution: KAIST Why notable: Co-author, bringing academic rigor from a top Korean university. His collaboration with NVIDIA researchers bridges academic RL theory with industrial robotics applications.

5. Operating Insights

Fixed Visual Encoders are Critical for Continual Learning

When you update a robot foundation model with new demonstrations, its internal visual representations shift. If your RL system relies on those internal representations, the knowledge it inherited from previous tasks breaks. ACCRUE solves this by using a frozen, external visual encoder (DINOv3). The paper found that "Replacing the fixed DINOv3 representation with features extracted by the stage-updated GR00T VLM yields worse retention and forward transfer... it supports the practical advantage of a fixed visual interface for inheriting the actor and critic across base-policy updates." (Sec 5.4) CTOs should ensure their continual learning pipelines decouple the visual encoder from the updatable foundation model.

Reference-Action Conditioning Prevents Relearning Full Skills

Instead of having the lightweight RL network learn a task from scratch, ACCRUE conditions it on the "reference action" proposed by the base foundation model. The RL network only learns a small refinement on top of that proposal. "Because the base policy already encodes task semantics and provides a task-level action proposal, the lightweight actor learns a local refinement conditioned on ã rather than reconstructing the full skill." (Sec 4.1) This drastically reduces the search space for the RL algorithm, making real-world training feasible in minutes.

6. Overlooked Insights

Robustness to Base-Policy Forgetting

A major operational fear is that if the base foundation model forgets an old task during sequential fine-tuning, the whole system fails. ACCRUE's RL post-training acts as a safety net. Even when the base model's performance drops due to reduced demonstration replay, the RL layer recovers the lost capability. The authors note, "Reduced demonstration replay degrades both base-policy and post-RL performance relative to using all demonstrations. Nevertheless, ACCRUE mitigates forgetting and maintains new-task learning efficiency even when the base policy itself forgets." (Sec 5.4) This means operators can be less precious about perfectly curating the base model's training data.

Architecture Agnostic

The framework is not a hack that only works on a specific type of model. The paper proves this by applying ACCRUE to both Vision-Language-Action models (GR00T) and World Action Models (Fast-WAM). "In Table 7, we apply ACCRUE on Fast-WAM (39) to prove the effectiveness regardless of model architecture." (Sec 5.4) This means companies can adopt this continual RL framework regardless of which foundation model architecture they are currently betting on.