Reinforcement learning-based fuzzing technology

Abstract

Fuzzing is a common vulnerability detection method in the modern software testing, which triggers potential vulnerabilities in the target program by generating variable input. However, traditional methods have the disadvantage of low code coverage due to the blind mutation of samples. To mitigate the problem, we model the process of traditional fuzzing as the Markov decision process and take use of the reinforcement learning algorithm to guide the direction of each step in the process of mutation to improve the quality of samples and the efficiency of fuzzing. In this paper, we implemented a general fuzzing system called RLFUZZ based on the reinforcement learning, taking the edge coverage as reward and using DDPG algorithm to maximize it. Experimental results show that DDPG-based RLFUZZ achieves greater edge coverage than baseline random mutation on LAVA-M dataset.