Meteorologists’ interpretations of storm-scale ensemble-based forecast guidance

Abstract

During the 2017 Spring Forecasting Experiment in NOAA’s Hazardous Weather Testbed, 62 meteorologists completed a survey designed to test their understanding of forecast uncertainty. Survey questions were based on probabilistic forecast guidance provided by the NSSL Experimental Warn-on-Forecast System for ensembles (NEWS-e). A mix of 20 multiple-choice and open-ended questions required participants to explain basic probability and percentile concepts, extract information using graphical representations of uncertainty, and determine what type of weather scenario the graphics depicted. Multiple-choice questions were analyzed using frequency counts, and open-ended questions were analyzed using thematic coding methods. Of the 18 questions that could be scored, 60%–96% of the participants’ responses aligned with the researchers’ intended response. Some of the most challenging questions proved to be those requiring qualitative explanations, such as to explain what the 70th-percentile value of accumulated rainfall represents in an ensemble-based probabilistic forecast. Additionally, participants providing answers not aligning with the intended response oftentimes appeared to consider the given information with a deterministic rather than probabilistic mindset. Applications of a deterministic mindset resulted in tendencies to focus on the worst-case scenario and to modify understanding of probabilistic concepts when presented with different variables. The findings from this survey support the need for improved basic and applied training for the development, interpretation, and use of probabilistic ensemble forecast guidance. Future work should collect data for a larger sample size to examine the knowledge gaps across specific user groups and to guide development of probabilistic forecast training tools.