Comprehensive survey of continual learning

Abstract

Continual learning（CL），which is also known as lifelong learning，is a major concern in the field of machine learning. It refers to the capability of a system to learn new tasks while retaining knowledge gained from previous ones. CL has broad relevance in many real-world applications that require continual adaptation and learning such as autonomous driving，robotics，and medical diagnosis. CL succumbs to the catastrophic forgetting issue，which occurs when a model overwrites or significantly diminishes previously learned information while training on new tasks. This issue is prominent in traditional machine learning paradigms where models are trained on static datasets. The goal of CL is to develop models that exhibit plasticity，which allows them to adapt to new information，and stability，which ensures that previous knowledge is not lost. Various strategies have been proposed in the literature to address this challenge，which are comprehensively reviewed in this study. Broadly，CL techniques can be classified into those based on continual training and those based on prompts. Methods based on continual training can be further divided into three main categories：replay-based，regularization-based，and dynamic architecture methods. Replay-based methods are among the most popular and well-established techniques in CL. These methods aim to mitigate catastrophic forgetting by storing and replaying data from previously learned tasks. By revisiting samples from prior tasks during training on new tasks，the model is effectively reminded of past knowledge，which helps maintain accuracy on earlier tasks. Replay-based methods can be further categorized into two types：replay memory construction and replay memory utilization. Replay memory construction focuses on saving the replay memory，and methods can be categorized into three types depending on how knowledge is stored：storing raw data，building generative models，and storing data features to construct memory. Memory replay methods that store raw data directly save the original data from old tasks along with the corresponding labels. The raw data are available in various formats，such as images，videos，audio，or text. During training on a new task，the stored memory data from old tasks are trained together with the new task data，which helps prevent forgetting. Storing raw data for memory replay requires no additional operations for storage and utilization，and it maintains consistency with the training process of the original model. The method of building generative models has been influenced by the recent development of generative models，where representative samples of past knowledge are generated for replay. These methods can be classified based on the type of generative model used. In CL，the strength of generative models lies in their ability to generate high-quality synthetic data，which helps the model overcome catastrophic forgetting issues common in traditional methods. Ultimately，the long-term memory capacity of the model is significantly enhanced. The method of storing data features is chosen when data features provide a good representation of the original data. In situations where privacy protection and storage constraints are a concern，storing data features rather than raw data or additional models becomes a practical solution. Replay memory utilization focuses on effectively leveraging the stored samples to enhance the efficacy of replay. This approach includes techniques such as data augmentation，knowledge distillation，Bayesian methods，optimization，and gradient projection，as well as representation alignment and bias correction. Regularization-based methods aim to address catastrophic forgetting by introducing constraints on the parameters of the model，which prevents them from undergoing drastic changes when learning new tasks. This approach enforces stability by preserving the critical aspects of previously learned knowledge through penalizing large deviations in the parameters of the model. Regularization methods can be categorized into those based on Laplace approximation，task representation constraints，Bayesian regularization，and knowledge distillation. The Laplace approximation technique approximates the posterior distribution of the old task as a Gaussian distribution，which imposes constraints on the important parameters of the old tasks. This type of regularization method works by adding penalty terms to the loss function to restrict changes to the parameters related to previous tasks when training on new tasks. In doing so，it achieves balanced learning of new and old tasks，which reduces the risk of catastrophic forgetting. Task representation constraint-based methods utilize the historical model to generate representations of current samples while applying regularization constraints based on these historical representations. The core idea of Bayesian regularization is to use a Bayesian framework to update model parameters，which allows the retention of knowledge from previous tasks while learning new ones. By regularizing model parameters within the Bayesian framework，this approach effectively balances the learning of new and old tasks，which reduces catastrophic forgetting. By contrast，knowledge distillation-based regularization methods do not require additional storage of replay memories. Instead，they leverage stored historical models as teachers while using the current training samples as input to guide learning. Dynamic architecture methods address new tasks and knowledge by gradually adjusting the structure of the model. These methods dynamically modify the architecture or parameters of the neural network based on changes in input data and the demands of new tasks. Thus，they add or reallocate network resources to learn new knowledge without forgetting the old. Dynamic architecture methods ensure CL by automatically expanding the network，activating important parameters，and freezing irrelevant parts，which allow the model to adapt to new tasks while avoiding catastrophic forgetting. Dynamic architecture methods can be further divided into multi-expert and subnetwork structures，dynamic scarification and masking techniques，dynamic structural adjustment，and the learning of additional task-related modules. In recent years，pre-trained large models，such as Transformer-based architectures，have achieved remarkable success across various domains due to their ability to generalize effectively across tasks. These models，which are commonly pre-trained on vast amounts of data，have strong representational power and are increasingly being applied in CL scenarios due to their ability to learn new tasks with minimal forgetting. Pre-trained models can be used in CL through two primary strategies：fine-tuning- and prompt-based methods. Fine-tuning involves adapting a pre-trained model to a new task by updating some or all of its parameters. In the context of CL，fine-tuning can be performed by freezing certain layers of the pre-trained model and only updating specific parts，such as task-specific layers，to prevent the model from losing knowledge gained from earlier tasks. Another approach is to fine-tune the model using task-specific learning rates，which ensures that important parameters for previous tasks are modified less during the training of new tasks. Prompt-based methods represent a newer approach to leveraging pre-trained models for CL. Rather than adjusting the internal parameters of the model，prompt-based methods guide its behavior by designing and inputting task-specific prompts. These prompts serve as auxiliary information that helps the model focus on the relevant aspects of the new task without altering its underlying architecture or parameters. Despite the advancements in CL techniques，several challenges remain. An important concern is the scalability of CL methods，especially in real-world applications where the number of tasks and data diversity can be vast. Another issue is achieving an optimal balance between plasticity and stability，given that methods that overemphasize stability may limit the ability of the model to learn effectively from new tasks. Future research in CL is expected to focus on the integration of pre-trained large models with traditional CL techniques. It will explore novel architectural designs and optimization strategies that can better address the demands of complex，real-world tasks. Additional works are needed to develop CL methods that are computationally efficient，particularly for large-scale models，without sacrificing performance or flexibility. By combining the strengths of large models with classical CL techniques，the field is set to achieve considerable advancements in creating intelligent systems capable of lifelong learning in dynamic and ever-changing environments.