Mitigating catastrophic forgetting in lifelong learning: a hybrid architecture integrating neural ordinary differential equations with memory-augmented transformers

Abstract

Catastrophic forgetting remains a critical challenge in lifelong learning where neural networks severely degrade on previous tasks when learning new ones. This paper proposes the first systematic integration of neural ordinary differential equations (Neural ODEs) with memory-augmented transformers to address this limitation. Neural ODEs enable smooth representation learning through continuous-time dynamics, while memory-augmented transformers provide explicit knowledge consolidation through attention-based retrieval. We derive rigorous PAC-learning theoretical bounds characterizing the relationships between model capacity, task sequence length, and forgetting severity, complemented by an adaptive memory management strategy. Comprehensive experiments on Split CIFAR-100, Permuted MNIST, and CORe50 demonstrate substantial improvements: 24% forgetting reduction and 10.3% accuracy gain over state-of-the-art methods. This work provides principled foundations for scalable lifelong learning systems capable of continuous knowledge accumulation without destructive interference.