Alexander Koran
All projects

April 2024

Dream On: DreamerV3 for Continual Reinforcement Learning via Permanent-Transient Critic Decoupling

Adapting DreamerV3 to retain prior knowledge while learning a sequence of reinforcement-learning tasks.

Read report Watch presentation Download slides

Evaluation returns for Continual-DreamerV3, original DreamerV3, and original DreamerV3 with reset across five alternating Walker tasks
Performance across alternating Walker Stand and Walker Walk tasks. Continual-DreamerV3 adapts to the second task and retains prior task knowledge more consistently than the original model. Figure 1 from the report.

Project overview

In this collaborative COMP 579 project with Zihan Wang and Thomas Lai, we adapted DreamerV3 for continual reinforcement learning by splitting its critic into transient and permanent value functions. The transient critic tracks current information while the permanent critic retains longer-term knowledge. We evaluated Continual-DreamerV3 on DeepMind Control Suite Walker Stand and Walker Walk tasks that alternated every 100,000 training steps, under the simplifying assumption that task boundaries were known. Across these experiments, the decoupled model retained prior behavior and adapted to task changes more reliably than the original DreamerV3 baseline. Additional experiments found that selectively reinitializing the value function at a task transition was particularly effective.