Accuracy
When communicating the results of an analysis, one of the first questions anyone will ask you is if you calibrated/validated your model. Often, this is a loaded question for a number of reasons, but one of the most addressable ones is to provide them the accuracy metric of your choice. While not always the case, this generally involves comparing two rasters, a reference and a ground truth to create what is known as a “confusion matrix”. This combination of values can be mathematically reduced to a single value, and that singular number is all that’s usually reported.
This of course can get the box checked at the end of the day, but leaves quite a lot of room for improvement. One of the “easiest” ways to improve on this baseline though, is to continue to perform that accuracy assessment at regular intervals. The single marker in that time series may not mean much in isolation, but when you can show steady or incremental improvements in that metric over time, that lends confidence to the underlying process representations you are applying to your model.
The other game to play here is of course “which metric do you choose?” Much like an extremely accessible form of p-hacking, They take catchment aggregated p-bias and take that as the room temperature measurement. “If water model versions are making improvements in each iteration that is seen as the sign that these efforts are improving. That is misleading not only because they’ve hinged that outcome on a single metric with no spatial significance, but also because the calibration and validation data is treated as a monolitic and stable set of observations instead of a living collection of data points which have a seasonality and even event based distributions.
Defining my matrix
There are two common ways to enumerate a confusion matrix but the most common is to place the truth to the left and the prediction across the top. This creates the following classification matrix. Note that I like extending to the null cases, although those are rarely included in the suite of commonly deployed metrics.
| Truth Positive |
True Positive (A) |
False Negative (B) |
Null Positive (E) |
| Truth Negative |
False Positive (C) |
True Negative (D) |
Null Negative (F) |
| Truth Null |
Unverifiable Hit (G) |
Unverifiable Rejection (H) |
|
Common Metrics
Accuracy (ACC)
\(ACC = \dfrac{TP + TN}{P + N} = \dfrac{A + D}{A + B + C + D}\)
Producer’s Accuracy (PA) / True Positive Rate / Sensitivity / Recall / Hit Rate
In remote sensing, this measures how well real-world features are correctly classified.
\(PA = \dfrac{TP}{P} = \dfrac{A}{A + B}\)
User’s Accuracy (UA) / Precision / Positive Predictive Value (PPV)
In remote sensing, this measures how often a pixel classified on the map actually represents that category on the ground.
\(UA = \dfrac{TP}{PP} = \dfrac{A}{A + C}\)
Probability of False Alarm (FPR) / Fall-out / Type I Error
\(FPR = \dfrac{FP}{N} = \dfrac{C}{C + D}\)
False Alarm Ratio (FAR)
\(FAR = \dfrac{FP}{PP} = \dfrac{C}{A + C}\)
False Negative Rate (FNR) / Miss Rate / Type II Error
\(FNR = \dfrac{FN}{P} = \dfrac{B}{A + B}\)
False Discovery Rate (FDR)
\(FDR = \dfrac{FP}{PP} = \dfrac{C}{A + C}\)
False Omission Rate (FOR)
\(FOR = \dfrac{FN}{PN} = \dfrac{B}{B + D}\)
True Negative Rate (TNR) / Specificity / Selectivity
\(TNR = \dfrac{TN}{N} = \dfrac{D}{C + D}\)
Negative Predictive Value (NPV)
The counterpart to Precision/UA for the negative class.
\(NPV = \dfrac{TN}{PN} = \dfrac{D}{B + D}\)
Balanced Accuracy
\(\text{Balanced accuracy} = \dfrac{TPR + TNR}{2} = \dfrac{\dfrac{A}{A + B} + \dfrac{D}{C + D}}{2}\)
Prevalence
\(Prevalence = \dfrac{P}{P + N} = \dfrac{A + B}{A + B + C + D}\)
F1-Score
\(F_1 = \dfrac{2 \times PPV \times TPR}{PPV + TPR} = \dfrac{2 \times TP}{2 \times TP + FP + FN} = \dfrac{2 \times A}{2 \times A + B + C}\)
Matthews Correlation Coefficient (MCC)
\(MCC = \dfrac{TP \times TN - FP \times FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}} = \dfrac{A \times D - C \times B}{\sqrt{(A + C)(A + B)(D + C)(D + B)}}\)
Critical Success Index (CSI) / Threat Score
\(CSI = \dfrac{TP}{TP + FN + FP} = \dfrac{A}{A + B + C}\)
Gilbert Skill Score (GSS) / Equitable Threat Score (ETS)
A modified version of CSI that accounts for the number of hits that could be expected purely by chance (\(A_{ref}\)).
\(GSS = \dfrac{A - A_{ref}}{A + B + C - A_{ref}}\)
Where \(A_{ref} = \dfrac{(A + B)(A + C)}{A + B + C + D}\)
Cohen’s Kappa Coefficient (\(\kappa\))
Commonly used alongside Producer/User accuracy to measure agreement while accounting for chance.
\(\kappa = \dfrac{p_o - p_e}{1 - p_e}\)
Where \(p_o = ACC\) and \(p_e = \dfrac{(A+B)(A+C) + (C+D)(B+D)}{(A+B+C+D)^2}\)
Common metrics across H&H
The hydrologic and hydraulic communities have a select number of accuracy metrics and traditional workflows that they reach to when they try to quantify the skill or performance of a model.
| Bias |
Continuous |
Long term average error (dry/wet) - also forcings |
| RMSE / RSR Nash-sutcliffe |
Cont |
A set of standard variance estimators, sometimes applied to long(Q) to de-emphasize extremes |
| Correlation |
Cont |
Linear correlation coefficient (Pearson’s r) |
| Kling-Gupta efficiency |
Cont |
Rational decomposition of error components (linear correlation, variability bias, and mean bias) |
| Peak discharge |
event |
Ability of model to predict Qp for events |
| Stormflow error |
event |
Ability of model to predict Rp for events |
| Conditional |
event |
Occurrence of an event (TP,FP,TN,FN) |
| Peak timing |
event |
Ability of model to predict Tp for events |
| Hydrologic signature |
mixed |
Other information content in streamflow signal |
NSE
(Nash and Sutcliffe 1970)
A continuous time series metric (\(-\inf < NSE < 1\)) \(NSE = 1 - \dfrac{\sum_{t=1}^{t=T}(Q_{sim}(t)-Q_{obs}(t))^2}{\sum_{t=1}^{t=T}(Q_{obs}(t)-\overline{Q_{obs}})^2}\)
A value > 0 indicates that the model predicts the variance of the system (streamflow) better than the mean. This metric generally falls into the the class of “least squared optimization and can be decomposed to include bias and correlation terms (but not cleanly).
LNSE
\(LNSE = 1 - \dfrac{\sum_{t=1}^{t=T} ( log(Q_{flow}(t)) - log(Q_{pred}(t)) )^2}{\sum_{t=1}^{t=T}( log(Q_{flow}(t)) - log(Q_{mean}) )^2}\)
Kling-Gupta efficency
(Knoben et al. 2019) \(KGE= 1 - \sqrt{(r-1)^2 + (\alpha-1)^2 + (\beta-1)^2}\) where \(r\) is the lienar corrolation between observations and simulations, \(\alpha\) is a measure of the flow variability error, and \(\beta\) a bias term \(KGE= 1 - \sqrt{(r-1)^2 + (\dfrac{\sigma_{sim}}{\sigma_{obs}}-1)^2 + (\dfrac{\mu_{sim}}{\mu_{obs}}-1)^2}\)
Note: Do not equate KGE with NSE
In the case where NSE = 0 (your model offers the same skill as the mean)
\(KGE= 1 - \sqrt{(0-1)^2 + (\dfrac{0}{\sigma_{obs}}-1)^2 + (\dfrac{\mu_{sim}}{\mu_{obs}}-1)^2}\)
\(KGE= 1 - \sqrt{(0-1)^2 + (\dfrac{0}{\sigma_{obs}}-1)^2 + (1-1)^2}\)
\(KGE= 1 - \sqrt{(0-1)^2 + (0-1)^2 + (1-1)^2}\)
Thus the KGE score for a mean flow benchmark is \(KGE(\overline{Q_{obs}}) = -0.41\)
See (Mathevet et al. 2024)
Metrics matter
I don’t have many pet peeves but one of them is listening to a conversation that poorly deploys or interprets accuracy metrics. Much of that frustration comes because those conversation tend to poorly wield these otherwise powerful aggregate statistics in a way that might be inappropriate, or that the “blame” for the poor performance is pinned to a misunderstanding of how variations in the confusion matrix do or do not impact the resulting calculation, and arguably more importantly, the outcome of the use of the prediction. One of the simplest ways I can try and describe this is along the lines of a paper on snow cover accuracy I wrote many moons ago. In this, we are comparing accuracy of two measurements, one, a SNOTEL snow pillow and two, a MODIS snow cover pixel. In short, we took 20+ years of daily sensor and satellite observations and compared a snow cover presences/absence metric Taking the data from Table 3 of (Coll and Li 2018),
| Month |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
| A |
99,134 |
80,492 |
83,064 |
63,886 |
30,313 |
5,538 |
171 |
8 |
1,237 |
18,951 |
64,085 |
79,714 |
| B |
5,473 |
3,779 |
6,284 |
13,287 |
21,423 |
11,733 |
2,268 |
140 |
1,033 |
11,432 |
16,762 |
6,627 |
| C |
397 |
330 |
502 |
1,004 |
1,937 |
1,441 |
790 |
1,151 |
2,136 |
5,737 |
3,620 |
985 |
| D |
1,355 |
1,655 |
4,407 |
16,302 |
59,356 |
130,360 |
179,941 |
174,369 |
166,161 |
108,457 |
25,111 |
3,292 |
| E |
163,031 |
159,109 |
172,667 |
152,530 |
98,668 |
25,731 |
1,586 |
172 |
3,415 |
44,574 |
125,803 |
175,798 |
| F |
1,839 |
1,827 |
4,366 |
15,548 |
59,702 |
88,136 |
87,754 |
97,594 |
91,506 |
81,879 |
27,115 |
4,859 |
| AC |
0.945 |
0.952 |
0.928 |
0.849 |
0.793 |
0.912 |
0.983 |
0.993 |
0.981 |
0.881 |
0.814 |
0.916 |
| AA |
0.370 |
0.332 |
0.322 |
0.305 |
0.330 |
0.517 |
0.661 |
0.638 |
0.630 |
0.470 |
0.340 |
0.306 |
| MCC |
0.374 |
0.487 |
0.580 |
0.642 |
0.612 |
0.469 |
0.104 |
0.017 |
0.438 |
0.621 |
0.604 |
0.471 |
| A AB |
0.948 |
0.955 |
0.930 |
0.828 |
0.586 |
0.321 |
0.070 |
0.054 |
0.545 |
0.624 |
0.793 |
0.923 |
| D DC |
0.773 |
0.834 |
0.898 |
0.942 |
0.968 |
0.989 |
0.996 |
0.993 |
0.987 |
0.950 |
0.874 |
0.770 |
We can see that accuracy, when divorced from context and across a highly imbalanced dataset, is a measure of prevalence, not predictive skill. Consider the months of July and August. If you look talk about the Clear Sky Accuracy of an observation, the performance appears flawless at 98.3% and 99.2% respectively. However, when we consider the snow cover counts for August. There are only 8 instances where snow was actually present (\(A\)), compared to 174,369 instances where snow was absent (\(D\)). If you created a baseline model that simply predicted “no snow” every single day, it would achieve a 99.2% accuracy rate in August. This prediction requires zero skill and provides zero operational utility, yet the aggregate statistic rewards it heavily because the environment is overwhelmingly snow-free. This distortion is why secondary metrics like the Matthews Correlation Coefficient (MCC) are necessary. The MCC scales from -1 to +1, where 0 represents the performance of a random guess: In April, during the shoulder season when snow presence and absence are more balanced, the MCC reaches 0.642, indicating genuine predictive capability. In August, as snow disappears from the landscape, the MCC drops to 0.017. While the accuracy metric suggests the model is performing at its best in August, the MCC reveals the truth: the model has lost its predictive utility. It is not successfully identifying snow; it is merely being carried by the statistical weight of a snowless summer. When evaluating model performance, high aggregate accuracy can mask a total failure to predict rare events. If a simple baseline heuristic can match your model’s accuracy, the metric is describing the background climatology, not the capability of the sensor.
A note on accuracy in the aggregate
Klotz et al. (2024)
References
Coll, James, and Xingong Li. 2018.
“Comprehensive Accuracy Assessment of MODIS Daily Snow Cover Products and Gap Filling Methods.” ISPRS Journal of Photogrammetry and Remote Sensing 144 (October): 435–52.
https://doi.org/10.1016/j.isprsjprs.2018.08.004.
Klotz, Daniel, Martin Gauch, Frederik Kratzert, Grey Nearing, and Jakob Zscheischler. 2024.
“Technical Note: The Divide and Measure Nonconformity – How Metrics Can Mislead When We Evaluate on Different Data Partitions.” Hydrology and Earth System Sciences 28 (15): 3665–73.
https://doi.org/10.5194/hess-28-3665-2024.
Knoben, Wouter J. M., Jim E. Freer, and Ross A. Woods. 2019.
“Technical Note: Inherent Benchmark or Not? Comparing Nash–Sutcliffe and Kling–Gupta Efficiency Scores.” Hydrology and Earth System Sciences 23 (10): 4323–31.
https://doi.org/10.5194/hess-23-4323-2019.
Mathevet, Thibault, Nicolas Le Moine, Vazken Andréassian, Hoshin Gupta, and Ludovic Oudin. 2024.
“Multi-Objective Assessment of Hydrological Model Performances Using Nash–Sutcliffe and Kling–Gupta Efficiencies on a Worldwide Large Sample of Watersheds.” Comptes Rendus. Géoscience 355 (S1): 117–41.
https://doi.org/10.5802/crgeos.189.
Nash, J. E., and J. V. Sutcliffe. 1970.
“River Flow Forecasting Through Conceptual Models Part I — A Discussion of Principles.” Journal of Hydrology 10 (3): 282–90.
https://doi.org/10.1016/0022-1694(70)90255-6.
