Toward a true spatial model evaluation in distributed hydrological modeling: Kappa statistics, Fuzzy theory, and EOF-analysis benchmarked by the human perception and evaluated against a modeling case study

Abstract

The hydrological modeling community is aware that the validation of distributed hydrological models has to move beyond aggregated performance measures, like hydrograph assessment by means of Nash-Suitcliffe efficiency toward a true spatial model validation. Remote sensing facilitates continuous data and can be measured on a similar spatial scale as the predictive scale of the hydrological model thereby it can serve as suitable data for the spatial validation. The human perception is often described as a very reliable and well-trained source for pattern comparison, which this study wants to exploit. A web-based survey that is interpreted based on approximately 200 replies reflects the consensus of the human perception on map comparisons of a reference map and 12 synthetic perturbations. The resulting similarity ranking can be used as a reference to benchmark various spatial performance metrics. This study promotes Fuzzy theory as a suitable approach because it considers uncertainties related to both location and value in the simulated map. Additionally, an EOF-analysis (Empirical Orthogonal Function) is conducted to decompose the map comparison into its similarities and dissimilarities. A modeling case study serves to further examine the metrics capability to assess the goodness of fit between simulated and observed land surface temperature maps. The EOF-analysis unambiguously identifies a systematic depth to groundwater table-related model deficiency. Kappa statistic extended by Fuzziness is a suitable and commonly applied measure for map comparison. However, its apparent bias sensitivity limits it's capability as a diagnostic tool to detect the distinct deficiency. Key Points: Quantifying the human perception in respect to pattern comparisons Performance metrics for spatial evaluation of distributed hydrological models Identifying complex spatial model defects by including new data in the model evaluation