The Hidden Universe of Operationalizing Natural Experiments: Navigating Extreme Weather

Abstract

Natural or quasi-experimental research designs are based on exogenous variation, oftentimes requiring operationalization before use in statistical tests. For this, researchers regularly lack clear theoretical guidance and need to make threshold decisions. We show for three types of shocks—extreme weather, immigration exposure, and commodity price changes—that this leads to a vast menu of equally plausible measurements, which are used unsystematically in ensuing literatures. This hidden universe of operationalization decisions has consequences: Drawing on geocoded survey and weather panel data for Switzerland, we show that different measurements of extreme weather exposure exhibit vast statistical and geographical variation. Importantly, researchers can support arbitrary conclusions when using singular operationalizations, while, in our case, the whole set indicates robust null effects. These results highlight that measurement decisions induce relevant researcher degrees of freedom rarely made transparent. We consequently discuss how current practice could improve, concluding with a template and recommendations.