Optimal estimator for assessing landslide model performance

Abstract

The commonly used success rate (SR) in evaluating cell-based landslide model performance is based on the ratio of successfully predicted landslide sites over total actual landslide sites without considering the performance in predicting stable cells. We proposed a modified SR (MSR), in which the performance of stable cell prediction is included. The advantage of MSR is to avoid over- and under-prediction while upholding the stable sensitivity throughout all simulated cases. Stochastic analyses are conducted by using artificial landslide maps and simulations with a full range of performances (from worst to perfect) in both stable and unstable cell predictions. Stochastic analyses reveal mathematical responses of estimators to various model results in calculating performance. The Kappa method, which is commonly used for satellite image analysis, is improper for landslide modeling giving inconsistent performance when landslide coverage changes. To examine differences among SR and MSR in real model application, we applied the SHALSTAB model onto a mountainous watershed in Taiwan. Case study shows that stable and unstable cell predictions are inter-exclusive in SHALSTAB model. The optimal estimator should compromise landslide over- and under-prediction. According to our 4000 simulations, the best simulation generated by MSR projects 83 hits over 131 actual landslide sites while the unstable cells cover only 16% of the studied watershed. By contrast, despite the fact that the best simulation deduced from SR projects 120 hits over 131 actual landslide sites, this high performance is only obtained when unstable cells cover an incredibly high landslide cover (∼75%) of the entire water-shed exhibiting a significant landslide over-prediction.