SPoRT Embarks on a Project to Produce Stream Height Forecasts using Machine Learning

The use of various soil moisture parameters from the SPoRT-LIS for monitoring drought conditions and assessing flood risk has been ongoing for years, and has been demonstrated for efficacy at other collaborative offices.  The use of soil moisture output for drought analysis is relatively straight-forward.  However, the use of the data for assessing flood risk has always been a bit more complicated and has involved the development of significant thresholds of rainfall and soil moisture that lead to flooding based on forecaster experience.  This method lends itself to some degree of subjectivity and has always been less quantitatively robust than preferred.  Nevertheless, the SPoRT-LIS data have provided valuable information regarding the state of soil moisture and the potential for flooding in near real-time.  There are older posts on the blog that describe the application of the data for flood risk assessment.

Now, the SPoRT group is embarking on a new project, employing a machine learning technique to provide a more quantitative measure of the relationship between soil moisture values, rainfall and stream height.  This new methodology involves the use of a Long Short-Term Memory (LSTM) Network.  The LSTM model for a particular drainage basin can be trained using a history of SPoRT-LIS soil moisture values, gauge height observations, and precipitation from the Multi-Radar Multi-Sensor Quantitative Precipitation Estimation data set.  Using this training data set, the forecast model for each basin is then run in real-time using the most recent gauge height observations, SPoRT-LIS soil moisture at various depths, and quantitative precipitation forecasts (QPF).  For this initial version of the gauge height forecasts, we’re using QPF values from the GFS model and the WPC.  One of the advantages of this type of modeling is that the great majority of computational power is on the front-end to train the model, while running the model in real-time is computationally much less expensive than running a hydrologic model.  Thus, we can run multiple precipitation scenarios for any basin quickly in real time.  For example, for the 40+ basins in our initial evaluation, the amount of processing time needed to run each basin at 6-hourly time steps out to 5 days with two different precipitation schemes takes just about 5 minutes!

The SPoRT group is working with some of our collaborative NWS offices (Huntsville, Nashville, Morristown, Sterling) and the Lower Mississippi and Mid-Atlantic River Forecast Centers (LMRFC and MARFC, respectively) for this initial test and evaluation of the stream height forecasts.  Although SPoRT has produced models for several thousand basins in the Southeast CONUS domain, this initial evaluation will involve a sub-set of streams, shown in the image below.

Image 1.  Gauge locations (black dots) for the initial evaluation of real-time gauge height forecasts from NASA SPoRT.

So, one might be asking…why is SPoRT engaged with stream height forecasting?  It’s important to remember that the SPoRT paradigm involves working closely with collaborative partners and assessing forecaster needs.  One of those needs involves having a better sense of flood risk at mid to long timescales during the 7-day forecast period.  Let us explain.  Operational gauge height forecasts from the RFC may not incorporate precipitation into the hydrologic models beyond one or two days due to operational and model limitations and constraints.  However, this can be problematic if an area is expecting heavy rainfall in the period beyond a day or two.  Take for example the current flooding event occurring across parts of the Tennessee Valley.  Operational gauge height forecasts for the Flint River (at Brownsboro, AL) from the afternoon of February 3rd indicated no rise forecast for the river (Image 2).  This is because the hydrologic models were not incorporating precipitation into the models as it was before the 48 hour cutoff.

Image 2.  Graph of gauge height observations (dotted blue line) and forecasts (dotted red line) for the Flint River at Brownsboro.  The forecast was valid approximately 1440 UTC 3 Feb 2020.  Observations are current through about 22 UTC 5 Feb 2020.  Horizontal bars at the top of the image indicate flooding thresholds (yellow=Action Stage, orange=Minor Flood, red=Moderate Flood)

However, heavy rain was expected in the region, which would certainly lead to some river rises.  The question is…how much?  Our old rules of thumb would have suggested flooding likely, based on soil moisture values and expected rainfall.  But again, the old rules didn’t indicate the time frame for flooding or the degree of flooding…it was generally just a qualitative “likely” or “not likely”.  So, to help alleviate this gap in knowledge, the new methodology provides objective, deterministic forecasts of stream or gauge height.  The image below shows the gauge height forecasts from the SPoRT LSTM models valid at about the same time.  Notice that the forecasts based on both GFS and WPC QPF scenarios indicated flooding was likely, while the higher precipitation from the GFS suggested flooding would reach Moderate Stage.  So, it’s easy to see here one of the advantages this type of modeling can have for overall hydrologic forecasting and situational awareness for the threat of flooding.

Figure 3.  SPoRT LSTM gauge height forecasts for the Brownsboro River at Brownsboro.  The black line shows observations, up to analysis time at 12Z 3 Feb 2020.  The blue dashed line contains gauge height forecasts based on GFS QPF, while the red line contains forecasts based on WPC QPF.  The blue vertical bars indicate 6-hourly GFS QPF, while the red bars indicate 6-hourly WPC QPF.

This post has become rather long.  So, we’re going to leave it here for now.  We’ll be providing more information about this project and discussing other advantages and limitations of our stream height forecasts in some upcoming posts as we continue this evaluation over the next couple of months.

– Kris and Andrew

Wide disparity in soil wetness this summer across Alabama and the Southeastern U.S.

The pattern of soil moisture across the state of Alabama and more broadly the Southeastern United States has evolved into one of marked disparity over relatively short distances (see Fig. 1d) and time frames. The SPoRT Center manages its own instance of the NASA Land Information System (i.e., “SPoRT-LIS”), which produces real-time soil moisture estimates in an observations-driven modeling framework.  Hourly and daily output fields are available on the SPoRT Center web page, and 3-hourly and daily data are delivered to select NOAA/NWS weather forecast offices for enhanced decision support in areas such as drought and hydrologic applications.

In general, 2019 has been quite a wet year across large portions of the U.S., with parts of Alabama being no exception (especially northwestern Alabama).  Beginning in May, a rapid deterioration in soil wetness occurred across many parts of the Southeast due to unusually hot and dry conditions during the last half of May through early June.  However, the latter part of June into July featured well above-average rainfall in some areas that reversed the rapid drying trends, especially over far northwestern Alabama, Mississippi, and western Tennessee.  The variations in SPoRT-LIS total-column soil moisture percentiles over the past 4 months are given in Fig. 1, illustrating the regional spatiotemporal trends described above from April through July.


Fig 1. Monthly-evolving, total-column SPoRT-LIS soil moisture percentiles (relative to a 1981-2013 climatology) over the Southeastern U.S., valid (a) 30 Apr, (b) 31 May, (c) 30 Jun, and (d) 31 Jul 2019. [Click on image for full resolution]

Interestingly, the soil moisture percentiles across far northern Alabama diminish quite substantially from west to east by the end of July (Fig. 1d), approaching the 98th percentile in western Lauderdale county (NW Alabama) to less than the 10th percentile across Jackson county (NE Alabama; counties of northern Alabama shown in Fig. 2).  The current conditions on 31 July 2019 relative to the 31 July historical soil moisture distributions from a 1981-2013 SPoRT-LIS daily climatology further illustrate this sharp zonal contrast in soil wetness anomalies within the county-based histograms of Fig. 3. The county-mean 0-2 meter relative soil moisture on 31 July 2019 in Lauderdale county is at the 87th percentile compared to the 33-year historical distribution of 31 July values (Fig. 3a).  Meanwhile, Limestone county to its east has a mean soil moisture at the 61st percentile (Fig. 3b), followed by the 51st percentile in Madison county (Fig. 3c) and only the 24th percentile in Jackson County, AL.  These results serve to illustrate the highly variable nature of rainfall and resulting soil wetness and agricultural impacts that can occur across the Southeastern U.S. during the summer months.


Fig 2. Counties of northern Alabama. The far northern counties of Lauderdale, Limestone, Madison, and Jackson are highlighted within the discussion text and in Figure 3.


Fig. 3.  Historical distributions of 0-2 meter relative soil moisture on 31 July and present-day county means on 31 July 2019 for all SPoRT-LIS grid points within a specific county, valid for far northern Alabama counties ranging west to east from (a) Lauderdale, (b) Limestone, (c) Madison, and (d) Jackson.  Gray bars represent the frequency distributions of 1981-2013 soil moisture values, vertical colored lines are reference percentiles, and the black dashed lines are present-day, county-averaged soil moisture value, with values tabulated in the upper-right of each panel. [Click on image for full resolution]

Finally, despite the month-to-month swings in soil moisture anomalies across much of the Southeast in recent months, one corridor that has persistently experienced abnormally dry conditions extends from southeastern Alabama into southern and central Georgia and western South Carolina. In fact, since 31 May (Figs. 1b-d), southeastern Alabama has seen soil moisture percentiles less than 20%, analogous to moderate to severe (or worse) proxy drought categories based on community-accepted conventions of percentile anomalies.  These dry regional pockets in the SPoRT-LIS analysis strongly correspond to the most recent U.S. Drought Monitor weekly product, issued on 30 July (Fig. 4).


Fig 4.  U.S. Drought Monitor weekly product valid for the week of 30 July 2019.


SPoRT LIS Shows Dry Soils During High Plains Blowing Dust Event…

Yesterday while working on some Dust RGB related training materials, I was looking at the RGB in AWIPS and noticed a dust event unfolding in real-time in the central High Plains.  The loop below shows Dust RGB imagery, generated by GOES-East, yesterday, 28 Jan 2019 during the late morning and early afternoon hours.  The loop is centered over NE Colorado and SW Nebraska where you’ll see the blowing dust develop and spread southeastward.  In case you’re not too familiar with this type of imagery, the dust is represented by the magenta colors.  It’s also possible to observe some of the individual dust streaks or plumes within the larger blowing dust event, which help to show their locations of origin.  (By the way, sorry about the loss of image fidelity when saving from AWIPS to an animated GIF).

Image 1.  GOES-East Dust RGB imagery, approx. 1737-2002 UTC, 28 Jan 2019. The blowing dust is defined by the magenta colors, near the center of the imagery.

Research has shown that it takes the right combination of factors to loft dust particles sufficiently to generate these larger scale blowing dust events, partly based on soil moisture and winds.  The SPoRT LIS 0-10 cm volumetric soil moisture (VSM) analysis at 18 UTC indicated very low values in the blowing dust source region, with VSM percentages generally around 12-16% (Image 2).  The METAR observations also indicate sustained winds were 35-40 knots with stronger gusts over 40 knots at one locations in the area.

Image 2. SPoRT LIS volumetric soil moisture (background colors) overlaid with surface METAR plots (yellow figures), valid at 18 UTC, 28 Jan 2019.

This last image is a snapshot of the Dust RGB taken at 1902 UTC, overlaid with surface visibility and ceiling observations.  Notice that at station KHEQ in far northeastern Colorado, a ceiling of 100 ft and visibility of 7 SM was reported, which was likely due to the blowing dust.

Image 3. GOES-East Dust RGB and ceiling and visibility observations from ground observation stations at approximately 19 UTC, 28 Jan 2019.

Some SPoRT collaborative NWS offices in the West CONUS have utilized LIS VSM values to locate areas where the probability of blowing dust events is heightened under the proper conditions.  However, SPoRT is looking into opportunities to better predict where these events will occur.

SPoRT LIS Shows Low Soil Moisture Conditions Near Large N. Cal Fire…

Just making a quick post here as I noticed there were relatively dry soil moisture conditions at the site of a rather large fire that developed quickly in Butte County, CA today.  The first image is the Fire Temperature RGB from the GOES-16 satellite (mesoscale domain sector 1) at 2018 UTC, today, 8 Nov 2018.  In this RGB, the fire can be observed by  colors ranging from near red to near white, just east of Chico, CA.  Notice there are a few white pixels, indicating relatively high emissions from shorter wavelengths (1.61 µm), and thus, relatively hot fire temperatures.


Meanwhile, soil moisture data from SPoRT’s Land Information System show low soil moisture percentiles (from the 33-year climatology, next image below) at the fire’s location east of Chico.  In fact, these values are  below the 2nd percentile at the fire’s location.


Lastly, the one year change in deep layer soil moisture values (0-200 cm) also show significant decreases in soil moisture centered at the fire’s location and especially just east over the last year.


SPoRT is conducting research and working closely with members of the wildfire community in the western U.S. to transfer these and other data sets for operational decision-makers.

-Kris W.

Dramatic Soil Moisture Transformation over North Carolina Associated with Flooding Rainfall from Hurricane Florence

Dramatic Soil Moisture Transformation over North Carolina Associated with Flooding Rainfall from Hurricane Florence

As anticipated, Hurricane Florence resulting in monumental rainfall totals, particularly across southern and eastern North Carolina.  This past week’s rainfall totals are depicted in Figure 1, derived from the NOAA/National Weather Service Advanced Hydrologic Prediction Service (AHPS).  Widespread totals exceeded 10” across most of southern/eastern North Carolina and far eastern South Carolina, with maximum rainfall of more than 20” along and within a few counties of the Atlantic Coast.


Figure 1.  Weekly total rainfall (inches), valid 11-18 September 2018, from the National Weather Service Advanced Hydrologic Prediction Service (AHPS) product.  Four counties are denoted, for which soil moisture histogram animations are shown later in this article.

The extreme rainfall dramatically impacted the soil moisture, which underwent a substantial transformation from very dry to near-saturation across south-eastern North Carolina.  Figure 2 shows soil moisture retrievals before and after Hurricane Florence from NASA’s Soil Moisture Active Passive (SMAP) mission, which estimates near-surface soil moisture (~top 5 cm) in near-real-time derived from passive microwave satellite observations.  A 6-day animation of hourly SPoRT-LIS simulated 0-100 cm relative soil moisture images overlaid with Multi-Radar Multi-Sensor rainfall contours (Fig. 3) nicely shows how the predominantly very dry soils across North and South Carolina prior to Florence were quickly moistened to near saturation over just a few days.  [Ongoing research at SPoRT seeks to further improve the experimental soil moisture estimates by assimilating SMAP retrievals into the SPoRT-LIS framework.]


Figure 2.  NASA Soil Moisture Active Passive (SMAP) Level 2 soil moisture retrievals from before (top panel; valid 11 September) and after Hurricane Florence (bottom panel; valid 16 September).


Figure 3.  Animation of hourly SPoRT-LIS 0-100 cm relative soil moisture images overlaid with MRMS precipitation contours, valid for the period 0000 UTC 12 Sep to 2300 UTC 17 Sep 2018. [Click on image for full resolution]

Similar to that shown in a companion blog article, Figure 4 shows the evolution of shallow (0-10 cm) to total column/deep (0-200 cm) soil moisture percentiles relative to the SPoRT-LIS 1981-2013 climatological database.  Anomalously dry soil moisture is depicted by orange/red colors, while anomalously wet soil moisture is given by green/blue colors.  Prior to Hurricane Florence, much of South Carolina and southern parts of North Carolina were experiencing unusually dry soil moisture for this time of year.  Despite the capacity for the soils to receive moisture, the historic rainfall was enough to overcome soil moisture deficits, quickly leading to near-saturated soil conditions in all model depths, and ultimately substantial flooding.  An interesting feature to note after the storm impact is the very high soil moisture percentiles greater than 98th percentile across South Carolina in the shallow layers.  Meanwhile, the deeper soils experienced excessive soil moisture percentiles above the 98th percentile predominantly over North Carolina where the heaviest rainfall occurred and where the pre-storm dry anomalies were not as large as in South Carolina.

Fig4a_vsm0-10percent_20180910-17_nc_animFig4b_vsm0-40percent_20180910-17_nc_animFig4c_vsm0-100percent_20180910-17_nc_animFig4d_rsm02percent_20180910-17_nc_animFigure 4.  Daily animations of SPoRT-LIS soil moisture percentiles relative to 1981-2013 climatology, valid from 10 to 17 September over model depths at (top image) 0-10 cm, (2nd image) 0-40 cm, (3rd image) 0-100 cm, and (bottom image) total model column 0-200 cm.  [Click on each individual image for full resolution]


Finally, the dramatic transformation in soil moisture is nicely highlighted by examining the present-day, county-averaged values relative to the 1981-2013 climatological distributions, as shown in Figure 5 at four select counties in North Carolina.  Robeson and Cumberland counties (first and third images in Figure 5) experienced the driest soils prior to Hurricane Florence (westernmost counties in Fig. 1), whereas New Hanover and Craven counties (second and fourth images in Figure 5) were more moist prior to Florence’s rainfall.  Each of the four sampled counties ultimately experienced record daily soil moisture values by 17 September (99.9th percentiles).  However, the moist antecedent soils in Craven county led to record soil moisture values by 15 September, whereas the other counties that began with drier soils achieved record values by 16 or 17 September.  In summary, despite predominantly dry soils prior to Hurricane Florence across much of the Carolinas, the tremendous 10-30”+ rainfall totals led to a quick saturation of the soils and massive, widespread flooding.

Fig5a_Robeson_County_NC_7dayloop_ending_20180917Fig5b_New_Hanover_County_NC_7dayloop_ending_20180917Fig5c_Cumberland_County_NC_7dayloop_ending_20180917Fig5d_Craven_County_NC_7dayloop_ending_20180917Figure 5. Daily animations of SPoRT-LIS total column, county-averaged soil moisture histograms, valid from 10-17 September 2018 for (top image) Robeson county, NC [city of Lumberton], (2nd image) New Hanover county, NC [city of Wilmington], (3rd image) Cumberland county, NC [city of Fayetteville], and (bottom image) Craven county, NC [city of New Bern].  Gray bars represent frequency distribution of 1981-2013 climatological 0-200 cm relative soil moisture values, vertical colored lines are reference percentiles, and black dashed line is present-day, county-averaged soil moisture value. [Click on each image for full resolution]


Hurricane Florence to Impact the Carolinas with Massive Rainfall

All eyes are on North and South Carolina as Major Hurricane Florence approaches the region over the next two days.  One important component to the official forecast is for the storm to slow down as it approaches the coast, due to the collapse of major atmospheric steering currents.  As a result, the NCEP Weather Prediction Center is predicting extreme rainfall amounts, especially for southeastern coastal North Carolina where 15-20”+ of rainfall is anticipated over the next 7 days (Fig. 1).


Figure 1.  NCEP Weather Prediction Center 7-day rainfall forecast, valid for the period 1200 UTC 12 September through 1200 UTC 19 September 2018. [Click image for full view]

An examination of the antecedent soil moisture is helpful to qualitatively assess the ability of the ground to absorb some of the moisture from the incoming rainfall.  Figure 2 shows a collage of shallow to deep soil moisture percentiles from 12 September within the four layers of the Noah land surface model, as being run in real time within NASA SPoRT’s configuration of the Land Information System (i.e., “SPoRT-LIS”).  The percentiles are derived from a 1981-2013 database of SPoRT-LIS daily soil moisture values in order to compare the present-day soil moisture to historical values on any given day of the year.  In Fig. 2, we see that recent soil moisture values are historically quite dry over central/northern South Carolina and into far southern North Carolina, with values under the 10th percentile (and even 2nd percentile, yellow/red shades) in some areas.  Meanwhile, as one traverses inland and northward, the soils steadily moisten to anomalously wet conditions (green/blue shades), especially over interior North/South Carolina to the Appalachian Mountains.


Figure 2.  SPoRT-LIS soil moisture percentiles on 12 September 2018, relative to 1981-2013 daily climatological values for the following layers: (a) 0-10 cm (top model layer), (b) 0-40 cm (top two model layers), (c) 0-100 cm (top three model layers), and (d) 0-200 cm (all four model layers). [Click image for full view]

The dry soil moisture anomalies near the coast suggest that the soils will initially be able to absorb incoming rainfall fairly effectively.  However, as prolonged heavy rainfall continues with the expected slow movement of Hurricane Florence, the soils should quickly become saturated, thereby leading to enhanced runoff and flooding potential over time.  So while having dry soils will be of some help early in the event, a prolonged exceptional rainfall up to 20”+ will lead to substantial flooding regardless of the initial soil moisture distribution.

The blog author documented a similar scenario (also over South Carolina), where substantial moisture from Hurricane Joaquin in Autumn 2015 led to 20”+ rainfall totals, largely occurring over dry soils in an area of moderate to severe drought, thereby completely eliminating the drought classification in South Carolina and producing substantial flooding.  A similar scenario was also seen associated with Hurricane Harvey in southeastern Texas last year, where very dry soils were prevalent prior to Harvey’s landfall north and west of Houston Metro.  However, given the very prolonged exceptional rainfall event, incredible soil moistening and flooding occurred anyway in much of southeastern Texas.


Wet streaks in soil moisture observed in west Texas…

This morning I observed some rather odd looking streaks in the 10.3 µm imagery in western portions of Texas.  Sampling the 10.3 µm data revealed alternating areas of relative warm/cool skin surface temperatures in the cloud free conditions in the area on the upstream side of departing deep convection.  The temperature difference in the skin temperatures were around 5 C at this time.  The 10.3 µm image below was taken from appox. 1357 UTC this morning (3 May 2018).

Image 1.  10.3 um imagery from GOES-16, 1357 UTC 3 May 2018

Arguably, the streaks of lower temperature values showed up better in the 3.9 µm imagery.  Notice the streaks or alternating bands of yellow/orange colors in portions of west Texas.

Image 2.  3.9 um imagery from GOES-16, 1357 UTC 3 May 2018

Realizing these temperature differences were likely due to the recent convective rainfall, I looked up the SPoRT LIS 0-10 cm volumetric soil moisture data, which corresponded nearly perfectly with the streaks of relative lower temperature values (Image 3).  So indeed, this was due to the recent heavy, convective rainfall across the area.

Image 3.  SPoRT LIS 0-10 cm Volumetric Soil Moisture, 15 UTC 3 May 2018