From COCOMO to GPT: A Comprehensive Evaluation of LLM-Based Software Effort Estimation

Abstract

Accurate software effort estimation remains a critical yet challenging task in software engineering. While recent advances in Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of software engineering activities, their effectiveness for effort estimation has not been systematically evaluated. This paper presents a comprehensive empirical benchmark comparing zero-shot LLM-based estimation with classical Machine Learning (ML) models and transformer-based regression approaches across two complementary settings: textual story-point estimation and structured project-level effort estimation. We evaluate multiple model families on two large-scale story-point datasets (Agile and NEO) and five widely adopted project-level datasets (Desharnais, COCOMO81, NASA93, Maxwell, and Kitchenham). Predictive accuracy is assessed across all datasets. The story-point benchmark further analyzes cross-project generalization, linguistic robustness, computational cost, and human–AI interaction effects, whereas the project-level benchmark evaluates robustness under noise injection, missing-value stress testing, and feature ablation. Experimental results show that off-the-shelf LLMs operating in zero-shot settings consistently underperform task-specific ML and transformer-based regression models in terms of absolute accuracy and cost efficiency. Classical ML models remain the most reliable estimators for structured datasets, while fine-tuned transformer regressors achieve the best performance on textual story-point estimation. Despite their lower numerical accuracy, LLMs demonstrate relatively low sensitivity to linguistic variation and may influence human judgment through persuasive explanations, as suggested by the exploratory human–LLM interaction study. These findings suggest that zero-shot LLMs may be suited as qualitative decision-support tools rather than as autonomous effort estimators. Overall, this work highlights the importance of architectural alignment, cost-aware evaluation, and human-centered considerations when deploying AI models for software effort estimation.