Large Language Models: Development in Model Scale and Challenges

Abstract

Since the 1950s, language modeling (Language Models, LMs) has been one of the primary approaches for tasks such as machine translation, as well as language understanding and processing. It has been widely applied in the field of natural language processing (Natural Language Processing, NLP), significantly improving the performance of tasks related to natural language understanding and generation. In recent years, large language models (LLMs) have made remarkable advancements in technical architecture and model scale, providing strong technological support for NLP and other fields. This paper presents a comprehensive review of the technological architecture and the development of the scale of large language models (LLMs), and delves deeply into the challenges these developments pose, along with the current strategies to address them. Finally, the paper summarizes and offers a prospective outlook on the future development directions of LLMs in terms of scale, providing insights and inspiration for the future development, training, and application of LLMs.