Promoting reproducibility and increased collaboration in electric sector capacity expansion models with community benchmarking and intercomparison efforts

Abstract

Electric sector capacity expansion models are widely used by academic, government, and industry researchers for policy analysis and planning. Many models overlap in their capabilities, spatial and temporal resolutions, and research purposes, but yield diverse results due to both parametric and structural differences. Previous work has attempted to identify some differences among commonly used capacity expansion models but has been unable to disentangle parametric from structural uncertainty. Here, we present a model benchmarking effort using highly simplified scenarios applied to four open-source models of the U.S. electric sector. We eliminate all parametric uncertainty through using a common dataset and leave only structural differences. We demonstrate how a systematic model comparison process allows us to pinpoint specific and important structural differences among our models, including specification of technologies as baseload or load following generation, battery state-of-charge at the beginning and end of a modeled period, application of battery roundtrip efficiency, treatment of discount rates, formulation of model end effects, and digit precision of input parameters. Our results show that such a process can be effective for improving consistency across models and building model confidence, substantiating specific modeling choices, reporting uncertainties, and identifying areas for further research and development. We also introduce an open-source test dataset that the modeling community can use for unit testing and build on for benchmarking exercises of more complex models. A community benchmarking effort can increase collaboration among energy modelers and provide transparency regarding the energy transition and energy challenges, for other stakeholders such as policymakers.