Deep Learning Representations of Programs: A Systematic Literature Review

Abstract

In the contemporary era, deep learning (DL) is increasingly recognized as a promising approach for enabling and optimizing various techniques, notably in the domain of DL for code (software programs). In essence, deep learning is mainly representation learning, which naturally holds for this domain. Thus, at the core of DL for code is deep representation learning for programs. The learned program representations can then be applied to various coding-related tasks, such as detecting vulnerabilities, providing recommendations for API usage, and extracting semantic and syntactic insights from extensive code lines. This is achieved by harnessing deep neural network architectures and deep-learning algorithms that take programs as inputs, serving various software engineering applications. In this article, we conduct a systematic literature search to review studies pertaining to the representation of programs using deep learning approaches and their corresponding applications. Our search yielded 178 primary studies published between 2017 and 2023. Through these studies in the latest literature, we provide a systematization of knowledge in deep learning representation of programs, concerning the raw inputs to the learning pipeline, neural network architecture employed, learning algorithm utilized, and downstream tasks (i.e., applications) of the learned representations. While examining the current landscape, we also identify limitations and challenges faced in the state-of-the-art, as well as promising future research directions in deep program representation learning.