Boosting Test Generation in Industrial Codebases: A Comparative Study of Base and Fine-Tuned LLMs

Abstract

Large Language Models (LLMs) have demonstrated promising capabilities in automated software engineering tasks, particularly in test case generation. Recently, researchers and practitioners have actively explored using open-source datasets with both open-source and commercial LLMs for generating test cases from source code and related software artifacts. However, the effectiveness of (fine-tuned vs base) open-source LLMs in industrial settings remains under explored. This study investigates the performance of various fine-tuned and base configurations of the DeepSeek-Coder 33B model (i.e., base and instruct) for unit test generation in a large-scale industrial code-base. This study propose a two-loop refinement strategy: Loop 1 generates initial test cases, while Loop 2 refines failed cases based on compilation errors and code coverage feedback. Eight different LLM configurations were evaluated using compilation success rate and line coverage as performance metrics. Our findings reveal that fine-tuned models significantly outperform base models, and the proposed refinement strategy leads to measurable improvements in both test compilation and code coverage. The use of fine-tuned-instruct model in loop 1 and fine-tuned-instruct model in loop 2 yielded the highest line coverage (62.56%) and compilation success. This work demonstrates the value of domain-specific fine-tuning and multi-phase feedback strategies in enhancing LLM-powered software testing pipelines.