Object-based verification of short-term, storm-scale probabilistic mesocyclone guidance from an experimental warn-on-forecast system

Abstract

An object-based verification method for short-term, storm-scale probabilistic forecasts was developed and applied to mesocyclone guidance produced by the experimental Warn-on-Forecast System (WoFS) in 63 cases from 2017 to 2018. The probabilistic mesocyclone guidance was generated by calculating gridscale ensemble probabilities from WoFS forecasts of updraft helicity (UH) in layers 2–5 km (midlevel) and 0–2 km (low-level) above ground level (AGL) aggregated over 60-min periods. The resulting ensemble probability swaths are associated with individual thunderstorms and treated as objects with a single, representative probability value prescribed. A mesocyclone probability object, conceptually, is a region bounded by the ensemble forecast envelope of a mesocyclone track for a given thunderstorm over 1 h. The mesocyclone probability objects were matched against rotation track objects in Multi-Radar Multi-Sensor data using the total interest score, but with the maximum displacement varied between 0, 9, 15, and 30 km. Forecast accuracy and reliability were assessed at four different forecast lead time periods: 0–60, 30–90, 60–120, and 90–150 min. In the 0–60-min forecast period, the low-level UH probabilistic forecasts had a POD, FAR, and CSI of 0.46, 0.45, and 0.31, respectively, with a probability threshold of 22.2% (the threshold of maximum CSI). In the 90–150-min forecast period, the POD and CSI dropped to 0.39 and 0.27 while FAR remained relatively unchanged. Forecast probabilities > 60% overpredicted the likelihood of observed mesocyclones in the 0–60-min period; however, reliability improved when allowing larger maximum displacements for object matching and at longer lead times.