Syntax-Aware Mutation for Testing the Solidity Compiler

Abstract

We introduce fuzzol, the first syntax-aware mutation fuzzer for systematically testing the security and reliability of solc, the standard Solidity compiler. fuzzol addresses a challenge of existing fuzzers when dealing with structured inputs: the generation of inputs that get past the parser checks of the system under test. To do so, fuzzol introduces a novel syntax-aware mutation that breaks into three strategies, each of them making different kind of changes in the inputs. Contrary to existing mutations, our mutation is able to change constructs, statements, and entire pieces of code, in a fine-grained manner that conforms to the syntactic rules of the Solidity grammar. Moreover, to explore new paths in the compiler’s codebase faster, we introduce a mutation strategy prioritization algorithm that allows fuzzol to identify and apply only those mutation strategies that are most effective in exercising new interesting paths. To evaluate fuzzol, we test 33 of the latest solc stable releases, and compare fuzzol with (1) Superion, a grammar-aware fuzzer, (2) AFL-compiler-fuzzer, a text-mutation fuzzer and (3) two grammar-blind fuzzers with advanced test input generation schedules: AFLFast and MO pt-AFL. fuzzol identified 19 bugs in total (7 of which were previously unknown to Solidity developers), while the other fuzzers missed half of these bugs. Also, fuzzol outperforms all fuzzers in terms of line, function, and branch coverage (from 3.75% to 408.8% improvement), while it is the most effective one when it comes to test input generation. Finally, our experiments indicate that our prioritization algorithm makes fuzzol explore new paths roughly one day (∼ 24 h) faster.