On the calculation and correction of equitable threat score for model quantitative precipitation forecasts for small verification areas: The example of Taiwan

Research output: Contribution to journalArticle

13 Citations (Scopus)

Abstract

As one of the most widely used skill scores for model quantitative precipitation forecast (QPF) verification and evaluation, the equitable threat score ETS differs from the threat score TS in that the random hits R are removed from its calculation. In practice, however, when applied to a set of verification points confined to small areas, the random assumption often becomes increasingly questionable or even invalid in larger events. As a result, the random hits are overestimated and the ETS becomes biased and not indicative of model skills. In this paper, such an issue is explored and demonstrated through the example of Taiwan with steep topography from Typhoon Morakot (2009) and mei-yu heavy-rainfall cases. It is found that the ETS is affected more seriously and scaled down by at least about 0.1 compared to the TS whenever the rain area occupies roughly 20% or more of the total verification area (if the random assumption of R is invalid). As such conditions often occur for small areas, it is vital to estimate R as correctly as possible for the ETS to work properly. A simple solution is offered by using all gridpoint values from the entire model domain, rather than just a small subset falling into the verification area, to estimate the random hit rate in the forecast. While the ETS remains unaltered in its definition, the proposed method yields the best estimates of R available by using the largest sample size from the model and subsequently better-behaved ETS values and is, therefore, recommended for all applications of ETS for small verification areas.

Original languageEnglish
Pages (from-to)788-798
Number of pages11
JournalWeather and Forecasting
Volume29
Issue number4
DOIs
Publication statusPublished - 2014 Jan 1

Fingerprint

typhoon
topography
rainfall
forecast
calculation
rain
evaluation
rate
method

ASJC Scopus subject areas

  • Atmospheric Science

Cite this

@article{5eae844a435e4b16a268e956a35131d7,
title = "On the calculation and correction of equitable threat score for model quantitative precipitation forecasts for small verification areas: The example of Taiwan",
abstract = "As one of the most widely used skill scores for model quantitative precipitation forecast (QPF) verification and evaluation, the equitable threat score ETS differs from the threat score TS in that the random hits R are removed from its calculation. In practice, however, when applied to a set of verification points confined to small areas, the random assumption often becomes increasingly questionable or even invalid in larger events. As a result, the random hits are overestimated and the ETS becomes biased and not indicative of model skills. In this paper, such an issue is explored and demonstrated through the example of Taiwan with steep topography from Typhoon Morakot (2009) and mei-yu heavy-rainfall cases. It is found that the ETS is affected more seriously and scaled down by at least about 0.1 compared to the TS whenever the rain area occupies roughly 20{\%} or more of the total verification area (if the random assumption of R is invalid). As such conditions often occur for small areas, it is vital to estimate R as correctly as possible for the ETS to work properly. A simple solution is offered by using all gridpoint values from the entire model domain, rather than just a small subset falling into the verification area, to estimate the random hit rate in the forecast. While the ETS remains unaltered in its definition, the proposed method yields the best estimates of R available by using the largest sample size from the model and subsequently better-behaved ETS values and is, therefore, recommended for all applications of ETS for small verification areas.",
author = "Chung-Chieh Wang",
year = "2014",
month = "1",
day = "1",
doi = "10.1175/WAF-D-13-00087.1",
language = "English",
volume = "29",
pages = "788--798",
journal = "Weather and Forecasting",
issn = "0882-8156",
publisher = "American Meteorological Society",
number = "4",

}

TY - JOUR

T1 - On the calculation and correction of equitable threat score for model quantitative precipitation forecasts for small verification areas

T2 - The example of Taiwan

AU - Wang, Chung-Chieh

PY - 2014/1/1

Y1 - 2014/1/1

N2 - As one of the most widely used skill scores for model quantitative precipitation forecast (QPF) verification and evaluation, the equitable threat score ETS differs from the threat score TS in that the random hits R are removed from its calculation. In practice, however, when applied to a set of verification points confined to small areas, the random assumption often becomes increasingly questionable or even invalid in larger events. As a result, the random hits are overestimated and the ETS becomes biased and not indicative of model skills. In this paper, such an issue is explored and demonstrated through the example of Taiwan with steep topography from Typhoon Morakot (2009) and mei-yu heavy-rainfall cases. It is found that the ETS is affected more seriously and scaled down by at least about 0.1 compared to the TS whenever the rain area occupies roughly 20% or more of the total verification area (if the random assumption of R is invalid). As such conditions often occur for small areas, it is vital to estimate R as correctly as possible for the ETS to work properly. A simple solution is offered by using all gridpoint values from the entire model domain, rather than just a small subset falling into the verification area, to estimate the random hit rate in the forecast. While the ETS remains unaltered in its definition, the proposed method yields the best estimates of R available by using the largest sample size from the model and subsequently better-behaved ETS values and is, therefore, recommended for all applications of ETS for small verification areas.

AB - As one of the most widely used skill scores for model quantitative precipitation forecast (QPF) verification and evaluation, the equitable threat score ETS differs from the threat score TS in that the random hits R are removed from its calculation. In practice, however, when applied to a set of verification points confined to small areas, the random assumption often becomes increasingly questionable or even invalid in larger events. As a result, the random hits are overestimated and the ETS becomes biased and not indicative of model skills. In this paper, such an issue is explored and demonstrated through the example of Taiwan with steep topography from Typhoon Morakot (2009) and mei-yu heavy-rainfall cases. It is found that the ETS is affected more seriously and scaled down by at least about 0.1 compared to the TS whenever the rain area occupies roughly 20% or more of the total verification area (if the random assumption of R is invalid). As such conditions often occur for small areas, it is vital to estimate R as correctly as possible for the ETS to work properly. A simple solution is offered by using all gridpoint values from the entire model domain, rather than just a small subset falling into the verification area, to estimate the random hit rate in the forecast. While the ETS remains unaltered in its definition, the proposed method yields the best estimates of R available by using the largest sample size from the model and subsequently better-behaved ETS values and is, therefore, recommended for all applications of ETS for small verification areas.

UR - http://www.scopus.com/inward/record.url?scp=84904972135&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84904972135&partnerID=8YFLogxK

U2 - 10.1175/WAF-D-13-00087.1

DO - 10.1175/WAF-D-13-00087.1

M3 - Article

AN - SCOPUS:84904972135

VL - 29

SP - 788

EP - 798

JO - Weather and Forecasting

JF - Weather and Forecasting

SN - 0882-8156

IS - 4

ER -