Large Language Models for Continual Relation Extraction

Abstract

Real-world data streams, such as news articles and social media posts, are dynamic and nonstationary, creating challenges for real-time structured representation via knowledge graphs, where relation extraction is a key component. Continual relation extraction (CRE) addresses this setting by incrementally learning new relations while preserving previously acquired knowledge. However, adapting models trained on stationary datasets to evolving data distributions remains a challenge. This work investigates the use of pretrained language models for CRE, focusing on large language models (LLMs) and the effectiveness of memory replay in mitigating forgetting. We evaluated decoder-only models and an encoder-decoder model on TACRED and FewRel in English. Our results show that memory replay is most beneficial for smaller instruction-tuned models (e.g., Flan-T5 Base) and base models such as Llama2-7B-hf. In contrast, the remaining instruction-tuned models in this work do not benefit from memory replay, yet some, such as Mistral-7B, already achieve higher accuracies without it and surpass prior methods. We further observed that Llama models in this work are more prone to schema-level errors (i.e., predictions beyond predefined relations). To the best of our knowledge, this work provides the first reproducible benchmarks Repository: https://github.com/sefeoglu/CRE_PTM for LLMs in CRE. It offers a novel analysis of knowledge retention and schema-level errors - dimensions that have not been systematically studied in earlier research.