What COVID-19 Taught Us About Observed Data vs. Model Projections: They Are Different - Let's Remember That When Interpreting Climate Models

COVID-19 - observed data on ICU cases and projected capacity

"All models are wrong, some are useful".  Predicting COVID-19 conditions has taught us that models come with a great deal of uncertainty, and are based on a lot of assumptions.  Furthermore, models have to be constantly updated over time with real observed baseline data to represent the starting point for future predictions. At least we recognize the difference between theoretical model projections and the past observations on COVID-19 conditions.  More attention should be given to the difference between theoretical models of climate effects and observed changes in extreme weather.

In early April, COVID-19 ICU cases were projected to increase to 1200 in a best case to about 1500 in a worst case in Ontario, increasing considerably from actual data counts in late March.  The chart at right shows that ICU beds peaked at under 300 cases by mid April, a fraction of the best case model prediction, and has declined since.  So model projections should be viewed with some caution, and the reliability of the projections should be questioned and validated where possible with real data.

Predicting future weather extremes due to climate change effects has a great deal of uncertainty as well.  The recurrence time of extreme rainfall is predicted to decrease due to climate change effects, meaning that the "return period" of storms would become smaller.  For example, a rainfall event that had a return period of 35 years today (meaning a probability of occurring in any year of 1/35, or 1 in 35) has been predicted to occur every 12 years in the future (i.e., a higher probability of happening each year of 1/12 or 1 in 12 ... that a greater chance than today's 1/35).  That is what is projected to occur in Canada from now to 2100.

The above example on decreasing recurrence times is from a simulation presented in Canada's Changing Climate Report by Environment Canada (link: https://changingclimate.ca/CCCR2019/).  It is for a future scenario with several assumptions about growth and emissions called the RCP8.5 scenario, representing a Representative Concentration Pathway of just one of several future scenarios.  The shift in 24-hour precipitation recurrence times are presented on Figure 4.20 b shown below:

Canada's Changing Climate Report Figures 4.20 b), Projected Extreme Precipitation Recurrence Time / Return Periods for Past, Present and Future Time Periods, RCP8.5 Model Simulation Scenario

As annotated above, today's recurrence time is noted as 35 years, the future recurrence time is 12 years and the past time was 50 years. So the model predicts these shifts in recurrence time (return period) and annual probability:

   Period         Recurrence Time       Probability Each Year
1986-2005             50 years                       2.0 %   (1/50)
2016-2035             35 years                       2.9 %   (1/35)
2081-2100             12 years                       8.3 %   (1/12)

Some have misinterpreted the theoretical, simulation model changes from past to present as 'actual' observed changes in extreme precipitation when in fact the Environment Canada report clearly notes these are 'projected changes' and are 'simulated by Earth system models' for the scenario RCP8.5.  A different scenario's simulated results, with different assumed emissions and growth, and different recurrence time shifts are presented in Figure 4.20 a) as well.

CBC News In Our Backyard - Flooding

CBC's In Our Backyard interactive notes "Climate change is no longer theoretical. It’s in our backyard" - unfortunately it presents theoretical past model trends as real changes that are "In Our Backyard" now.  Here is the online report link: https://www.cbc.ca/news2/interactives/inourbackyard/

CBC News report: "Climate change is making extreme rainfall a more frequent occurrence. Storms that historically happened only once every 50 years are now coming every 35 years or less. By the end of the century, they could happen once every 12 years on average, according to a recent climate report from Environment Canada. All this increases the potential for urban flooding."

CBC News In Out Backyard Extreme Rainfall Frequency - Past, Present, and Future Recurrence Times Confuses Simulation Model Projections With Observed, Historical Trends

So while predicted changes are only theoretical, CBC News mistakenly reports that changes have already occurred and are 'now coming' at smaller recurrence intervals (i.e., higher frequency and higher probability each year).

The CBC Ombudsman has indicated that the CBC should be careful to distinguish between past, present and future extreme rainfall trends, as noted in a recent post: https://www.cityfloodmap.com/2020/05/past-present-or-future-cbc-ombudsman.html

We agree.

A review of historical extreme rainfall trends in one region of Canada affected by may flooding events has shown no decrease in the recurrence time, or return period, of extreme precipitation.  A previous post showed that today's 35 year storms are actually occurring less frequently than in the past. In southern Ontario, long term climate station observations show that the average 25 to 50 year rainfall intensities today are actually slightly smaller than they were considering observations up to 1990. See previous post: https://www.cityfloodmap.com/2020/05/southern-ontario-extreme-rainfall.html

Analysis of the Version 3.10 Engineering Climate Datasets IDF Files updated in March 2020 show that southern Ontario long term rainfall intensities have decreased slightly since 1990, on average by 0.1%.  The 50 year return period rainfall intensities are on average unchanged.

If 50 year rainfall intensities actually occurred more frequently and now occur at a 35 year return period, as CBC mistakenly reported, then the magnitude of the 50 year intensities would have had to increase by about 6%.  This considers the example long term climate station at Toronto's Pearson International Airport - the 35 year 24-hour rainfall intensity of 99.7 mm at the airport would have to increase to the 50 year intensity of 105.7 mm.  Back in 1990, the 50 year 24-hour rainfall intensity at the airport was 109.3 mm, meaning the 50 year rainfall has decreased by several percentage points.  Here are the 1990 data (copied from my top desk drawer):

Toronto Pearson International Airport IDF Table With Data Up to 1990 - 50 year design rainfall intensity of 109.3 mm  (shown here) was higher than today's version 3.10 Engineering Climate Datasets intensity of 105.7 mm (see table below to 2017).  

Here are the recently updated IDF values from Environment Canada considering data up to 2017:

Toronto Pearson International Airport IDF Table With Data Up to 2017 - 50 year design rainfall intensity of 109.3 mm (see previous table to 1990) shown is lower than today's version 3.10 Engineering Climate Datasets intensity of 105.7 mm (shown here).

Climate models that predict more frequent future rainfall intensities, characterized by shorter recurrence times (i.e., lower return periods = higher probabilities of occurrence) are not necessarily in step with observations (see Toronto airport example above and previous post on southern Ontario long term stations).  Here is a comparison of past trends in 100-year rainfall intensity based on observed data and projections from various studies - the actual data curve is already 'flat', so the need to flatten the curve can only be made based on projections and not past data.

Extreme Rainfall IDF Trends - Toronto 24-Hour 100-Year Rainfall Volumes per Environment Canada Engineering Climate Datasets - Past Data and Linearly Projected Trends Shown in Black.  Various Studies and Models Project Significant Increases That Have Not Shown Up In The Data Observed Data Statistics

Just like COVID-19 models have considerable uncertainty and must rely on observational data to calibrate and validate them - so they they are more reliable and useful in making projections of the future - climate models require checks on accuracy and usefulness.  Media like CBC News may not discern between model predictions and actual trend data which can mischaracterize trends in extreme weather.  Since models predicting extreme rainfall do not appear to match past observations over the recent past few decades, the accuracy and reliability to project conditions over the next 80 years should be closely scutinized.

While in the case of COVID-19, the need for "flattening the curve" is clear given the close scrutiny of observed data that has shown rising counts of infections, hospitalizations or deaths - that gives clear direction on actions to be taken to mitigate observed phenomena.  In the case of COVID-19, these values may even increase at an exponential rate.  In contrast, the IDF curve trends are largely flat if not already declining based on observed data in some regions.  Any change in extreme rainfall trends has been explained by natural variations (i.e., trends can go up).

***

There is a long-standing gap in the media mixing up predictions of extreme weather and actual Environment Canada observed data trends - sometimes a single report can start a narrative that can go unchecked for some time.  The "Telling the Weather Story" report is one such example where a theoretical shift in extreme weather has been reported, and repeated endlessly in the media as actual data when it is clearly not:

Storm intensity not increasing - factual review of engineering data - Canada and Ontario from Robert Muir

Southern Ontario Extreme Rainfall Trends - Environment Canada Engineering Climate Datasets IDF Tables

Environment Canada's version 3.10 update to Canada's rainfall IDF tables and curves shows more increases than decreases and slightly more significant increases than decreases - overall trends were shown in a recent post: https://www.cityfloodmap.com/2020/05/annual-maximum-rainfall-trends-in.html.

Regional trends may be up or down and warrant further review.  In Southern Ontario the IDF intensities for long term stations have been reviewed and compared with pre-version 1.00 statistics up to 1990 (similar to version 3.00 and version 2.30 comparisons shared in earlier posts).  The following table shows average changes since 1990, and the surrounding arrows suggest how these changes may influence infrastructure design, if at all.

Southern Ontario IDF Rainfall Intensity Trend Table - Environment and Climate Change Canada's Engineering Climate Datasets, Pre-Version 1.00 (up to 1990) to Version 3.10 (up to  2017)

The 21 stations assessed include:  Sarnia Airport, Chatham WPCP, Delhi CS, Port Colborne, Ridgetown RCS, St Catharines Airport, St Thomas WPCP, Windsor Airport, Brantford MOE, Fergus Shand Dam, Guelph Turfgras CS, London CS, Mount Forest (Aut), Stratford WWTP, Waterloo Wellington Airport, Bowmanville Mostert, Hamilton Airport, Hamilton RBG CS, Oshawa WPCP, Toronto City, Toronto International Airport (Pearson).

The above changes in design rainfall intensities since 1990 do not suggest any overall shift that would affect how municipal drainage infrastructure would be designed considering current weather conditions.  That is, overall intensities have decreased by 0.2% which in negligible.  The 5-minute, 2-hour and 6-hour intensities decreased consistently across all return periods.  The change however is also negligible.  On average frequent intensities, e.g., 2-year intensities expected every couple of years, and 5-year intensities, used for storm sewer design showed overall decreases.  Again the changes are negligible.  The 100-year intensities increased by 0.1% overall which is also negligible, especially considering the confidence limits with such statistics and the uncertainty in curve fitting (Gumbel distributions are used, other distributions would provide shifts in results).  One would expect rare intensities to increase over time for skewed distributions given sampling bias with short records (i.e., limited observations of extreme events are expected to lead to underestimates of 100-year statistics).

Of course considerations must be made to account for future changes and uncertainties.  Some cities and regions have incorporated allowance for climate change effects.  In Quebec a 18% allowance is standardized.  Some cities (e.g., Ottawa) include a 20% stress test to evaluate any unacceptable conditions that warrant design changes to address future potential risks.  Others incorporate stress test hyetographs in the design process - the Windsor/Essex Ontario standards include a stress test event that has 39% greater volume than the standard 100-year design storm (NB - the daily volume is increased to account for that additional volume distributed uniformly across 24 hours, while peak hyetograph intensities are only nominally affected). 

The chart below shows the IDF trends at these long-term record Southern Ontario stations.

Southern Ontario IDF Rainfall Intensity Trend Chart by Duration - Environment and Climate Change Canada's Engineering Climate Datasets, Pre-Version 1.00 (up to 1990) to Version 3.10 (up to  2017)

It is clear that that the short duration intensities (red and orange bars representing 5 and 10 minute durations) have decreased the most as shown in the table above. The chart and table below shows a more simplified version of the above chart indicating the range of changes observed.

Southern Ontario IDF Rainfall Intensity Trend Chart by Duration - Environment and Climate Change Canada's Engineering Climate Datasets, Pre-Version 1.00 (up to 1990) to Version 3.10 (up to  2017)

The above chart shows that 2-year and 5-year IDF rainfall intensities have decreased most consistently among all stations.  Those intensity estimates benefit from many observations each year to determine the statistics.  Rare event intensities from 25-year to 100-year return periods have more equal increases and decreases yet the decreases are greater.  The magnitude of the changes, both increases and decreases are on average negligible.  Even the greatest increase of 1.2% from 1990 to 2017 (27 years) is negligible for the purpose of hydrologic analysis and drainage infrastructure design.

Toronto Area Extreme Rainfall Intensity Trends - Environment Canada IDF Curve Updates 2020

Environment and Climate Change Canada's intensity duration frequency (IDF) data describes the rare extreme rainfall intensities used to design drainage infrastructure and to assess river peak flows/flood flows when the big storms hit.

Some climate station IDF analysis have been due for an update for several years.  The new version 3.10 update in 2020 extends analysis to 2016-2017 in many cases.  The trends in short duration intensities can show how flood risks are changing due to changing climate and any more severe weather. Many municipalities and researchers have updated their IDF statistics internally and have reported trends in extreme rainfall intensities (see previous post for Ontario studies: https://www.cityfloodmap.com/2020/05/annual-maximum-rainfall-trends-in.html).

Earlier versions of the IDF datasets are available to characterize annual maximum rainfall, dating back to 1990.  The tables below shows updated trends in 100-year rainfall intensity over a short 5 minute duration at Buttonville Airport in Markham (updated in version 3.10 in 2020) and in Toronto and Mississauga (updated in version 3.10 in 2019).

For short durations, it appears that extreme rainfall intensities are decreasing in the Greater Toronto Area.  The rare 100-year intensities are decreasing from 4-7% at the three locations above.  The more frequent 2-year intensities are decreasing by about 5-8%.

There is higher uncertainty with 100-year storm intensities due the rarity and spatial distribution of events.  However, the more frequent 2-year intensities that rely on many more rainfall observations every year to characterize these average annual peaks are more reliable.  To illustrate this, consider the 95% confidence bands on the Environment Canada IDF 5-minute data for Buttonville Airport, in southern York Region, just north of Toronto:

Extreme Rain 95% Confidence Bands Buttonville Airport, Markham, Ontario - Environment Canada Engineering Climate Datasets v3.10

The 100-year 95% confidence band of 97 mm/hr (2 x 48.5) is 46% of the expected value of 210.8 mm/hr.  In contrast, the 2-year band of 22.6 mm/hr is only 22% of the expected value of 104.7 mm/hr, a relatively tighter band.  The tight band makes the observed decrease in 2-year rain intensities more noteworthy, i.e., the 2-year intensity has decreased 7.6% which is a significant proportion of the uncertainty band.

The longer duration annual maximum rainfall series for Toronto and Mississauga have relatively tighter confidence bands for both 100-year and 2-year intensities:

Extreme Rain 95% Confidence Bands Pearson International Airport, Mississauga, Ontario - Environment Canada Engineering Climate Datasets v3.10

Extreme Rain 95% Confidence Bands Toronto, Ontario - Environment Canada Engineering Climate Datasets v3.10

Given that rainfall intensities are decreasing over most durations in the Greater Toronto Area, based on Environment Canada's annual maximum series and derived 2-year and 100-year IDF design intensities, why does media often report a 'new normal' of more extreme weather?  This may be due to lack of familiarity with data and a tendency to exercise an 'availability bias', i.e., simple listing of recent events that have caused damages and association of those events with increasing rainfall intensities, but without checking the actual rainfall trends or investigating other factors.  For more on cognitive biases in framing and solving complex problems read about Thinking Fast and Slow on Floods and Flow:   https://www.chijournal.org/C449.   Given the proliferation of rainfall gauges across the GTA, many more than the Environment Canada numbers, the observation of many extreme events over a short duration may not be a statistical anomaly at all, as analysis here shows: https://www.cityfloodmap.com/2019/03/are-six-100-year-storms-across-gta-rare.html

The CBC has made several corrections to its stories on changing extreme rainfall trends in the past, sometimes finding violation of its Journalistic Standards and Practices regarding accuracy of reporting - here are some examples: https://www.cityfloodmap.com/2019/06/cbc-correcting-claims-on-extreme.html.  Most recently the CBC Ombudsman in a review entitled "Past, Present, or Future" wrote that "journalists could have been clearer with their choice of tenses" - more on that here: 

https://www.cityfloodmap.com/2020/05/past-present-or-future-cbc-ombudsman.html.

The CBC Ombudsman stated "There are appropriate distinctions made between observed phenomena and predicted phenomena" in its reporting noting this excerpt: 

• Precipitation will increase in much of the country.
• Weather extremes will intensify.

The last two bullet points are careful to use the future tense.

The recently updated Environment Canada IDF data in the GTA supports the CBC's perspective that intensified weather extremes are predicted phenomena as opposed to observed ones.