Diffusion of Forecasting Principles through Software

Abstract

Do forecasting software programs facilitate good practices in theselection, evaluation, and presentation of appropriate forecastingmethods? Using representative programs from each of four market categories,we evaluate the effectiveness of forecasting software in implementingrelevant principles of forecasting. The categories are (1) spreadsheetadd-ins, (2) forecasting modules of general statistical programs,(3) neural network programs, and (4) dedicated business-forecastingprograms. We omitted one important category�demand planning software�becausesoftware developers in that market declined to submit their productsfor review.In the aggregate, forecasting software is attending to about 50 percentof the basic principles of forecasting. The steepest shortfall occursin assessment of uncertainty: programs are often secretive abouthow they calculate prediction intervals and uninformative about thesources of uncertainty in the forecasts. For the remaining areasof evaluation�preparing data, selecting and implementing methods,evaluating forecast accuracy, and presenting forecasts�we rated thepackages as achieving 42 to 51 percent of the maximum possible ratings(the ratings assigned for best practices).Spreadsheet add-ins (16% of best-practices rating) have made rudimentaryregression tools and some extrapolative forecasting techniques accessibleto the spreadsheet analyst; however, they do not incorporate bestpractices in data preparation, method selection, forecast accuracyevaluation, or presentation of forecasts.Forecasting modules of general statistical programs (42% of best-practicesrating) provide effective data preparation tools; however, with theexception of one of these programs, they do not adequately help usersto select, evaluate, and present a forecasting method. To implementbest practices, the forecaster must perform macro programming andmultiple-step processing.Neural network packages (38% of best-practices rating) facilitatemany best practices in preparing data for modeling and in evaluatingneural network models. They do not use the more traditional modelsas comparative benchmarks, however, to test whether the neural netimproves accuracy enough to justify its added complexity and lackof transparency.Dedicated business-forecasting programs (60% of best-practices rating)have the best record for implementation of forecasting principles.Data preparation is generally good, although it could be more effectivelyautomated. The programs are strong in method selection, implementation,and evaluation. However, they lack transparency in their assessmentsof uncertainty and offer forecasters little help in presenting theforecasts. Three of the dedicated business-forecasting programs containfeatures designed to reconcile forecasts across a product hierarchy,a task this group performs so commendably it can serve as a rolemodel for forecasting engines in demand-planning systems.