ticket_370_TicketDetails - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki

Assess possible degradation of TC forecasts by ACCESS-G caused by not assimilating some observation types

Bureau currently does not receive, and so its models do not assimilate, a large number of observation types that UKMO does. This is thought to degrade Bureau's global model forecasts.

There is a view that the observation coverage is sufficient (Kelly et al. 2007).

However Fiorino (2009) showed that changes (in this case IFS physics) may not show up in gross verification statistics but can show up in tropics: e.g. tropical winds, tropical cyclone tracks.

Also the impact of one observation type on its own may not be clear. However the synergistic effect of a number of observation types can be magnified.

Additional observation types

Following two tables show a comparison of observations which are currently used operationally in ACCESS-G3 and those used in UKMO OS42,

In-situ observations

Obstype	Usage in ACCESS-G3	Usage in UKMO OS42

METAR, SYNOP	used	used
TEMP, PILOT, SONDE	used	used
AIREPS, AMDARS	used	used

Satellite observations

Obstype	Usage in ACCESS-G3	Usage in UKMO OS39	Comment

ATOVS	MetOp-1, MetOp-2, NOAA-15, NOAA-18, NOAA-19	MetOp-1, MetOp-2, NOAA-15, NOAA-18, NOAA-19
AIRS	Aqua	Aqua
IASI	MetOp-1, MetOp-2	MetOp-1, MetOp-2
CrIS	S-NPP	S-NPP
ATMS	S-NPP	S-NPP
AMSR	GCOM-W1	GCOM-W1
SSMIS	DMSP17, DMSP18	DMSP17, DMSP18
GMI	not used	GPM	UKMO OS40 started assimilating GMI from 20180213T06
MWTS-1, MWHS-1	not used	FY-3B, FY-3C	In OPS app config MetDB subtype is `ATOVS` for FY-3B and `MWTS` for FY-3C; Q. Does this mean MWHS is not used?
Satwind	METEOSAT-8,
METEOSAT-11,
H-8,
GOES-15,
GOES-16,

Aqua

| METEOSAT-8,

H-8, GOES-15,

MetOp-1, MetOp-2, NOAA-15, NOAA-18, NOAA-19, S-NPP, Aqua, Terra, mixture of GEO/LEO | ACCESS-G3 uses METEOSAT-11 and GOES-16 which are not used by UKMO; the control and the experiment do not use these 2 satellites | | Scatwind | MetOp-1, MetOp-2, CORIOLIS | MetOp-1, MetOp-2, CORIOLIS | | | GPSRO | MetOp-1, MetOp-2, TerraSAR-X,

Note 1. For satwind and surface obstypes I could not come up with apps that would allow reading of all obs. For this I compared the numbers given by print-obstore with numbers in job.stats

Experimental set-up

For each TC case we will use a control and a test. The control will assimilate all observation types including those that we currently do not assimilation but are used by UKMO; the test will use only the observations used operationally in ACCESS-G3.

In order to test whether the trial was set up correctly (in fact the first trial gave null result: i.e. the additional observations did not have much impact) I ran a trial which should show differences (sanity check).

We will mainly assess the difference in track forecasts between control and test. However a question arises as to how robust the difference in track forecasts is. It's possible that the difference arose because of the inherent chaotic nature of the data assimilation system and the forecast model. To allow us to approximately the effect of this randomness we will also create an initial condition by adding global, random perturbations to the control.

UM N320 on Raijin

For the trial period, 20180224T06-20180501T12 2 suites were set up:

u-bl688 - control
u-bm460 - experiment that excludes all observations which are not used by ACCESS-G3

For the trial period, 20190809T06-20190909T12 2 suites were set up:

u-bm976 - control
u-bn066 - experiment that excludes all observations which are not used by ACCESS-G3

UM N320 on Gadi

After migrating to Gadi I ran no new trials. However I prepared two suites in case I needed to repeat the trial,

u-bx821 - control
u-bx961 - experiment that excludes all observations which are not used by ACCESS-G3

ToDo for u-bq615

switch UKMO_MASS to /scratch
Change MPI launcher option -np back to -n; also all other suites that run on Gadi
Under the family suite-runtime-gl-A-nci.rc -> [VER_SHARED] -l jobfs=1GB may be needed
Under the family suite-runtime-gl-A-nci.rc -> [GL_ARCHIVE] PBS memory may need to be increased to 5 GB
In suite-runtime-gl-A-nci.rc you may need to adjust PBS resources defined by Jinja2 dictionary variable, OBS_TASKS_DICT

Trial periods

The following is a list of Western Pacific TC's used in this experiment and the trial periods,

| TC | First cycle of trial | last cycle of trial | lon/lat when first declared | Category | Comments | | --- | --- | --- | --- | --- | --- | |||||| First trial period 20180224T06-20180501T12 | | HOLA | 2018022406 | 2018031112 | -15.8, 169.0| mincp=952 Cat 4 | SouthPacific named on 20180306 and at its naming Cat 1 | | LINDA | 2018030306 | 2018032512 | -20.3, 159.0| mincp=994 | SouthPacific | | MARCUS | 2018030606 | 2018032512 | -10.3, 132.6| mincp=916 | AusTopEnd | | NORA | 2018030906 | 2018032712 | -9.9, 137.0| mincp=958 | AusTopEnd | | IRIS | 2018031406 | 2018040512 | -14.1, 160.5| mincp=987 | SouthPacific | | JELAWAT | 2018031406 | 2018040112 | 7.2, 138.7| mincp=935 | NorthWestPacific | | JOSIE | 2018032106 | 2018040212 | -18.4, 175.7| mincp=993 | SouthPacific | | KENI | 2018032906 | 2018041112 | -16.1, 170.1| mincp=970 | SouthPacific | | FLAMBOYAN | 2018041706 | 2018050112 | -8.6, 90.5| mincp=983 | SouthIndianOcean | | | | | | | | |||||| Second trial period 20190809T06-20190909T12 | | | | | | | | |LEKIMA | 2019080412 | 2019081212 | 17.5, 130.8| cpmin=920 |NorthWestPacific | |KROSA | 2019080612 | 2019081600 | 18.9, 142.4| cpmin=950 |NorthWestPacific | |BAILU | 2019082112 | 2019082512 | 15.8, 130.7| cpmin=985 |NorthWestPacific | |PODUL | 2019082700 | 2019083000 | 14.7, 127.0| cpmin=992 |NorthWestPacific | |LINGLING | 2019090200 | 2019090712 | 15.1, 126.1| cpmin=940 |NorthWestPacific | |KAJIKI | 2019090300 | 2019090300 | 15.9, 107.1| cpmin=998 |NorthWestPacific | |FAXAI | 2019090512 | 2019091000 | 20.6, 154.6| cpmin=955 |NorthWestPacific |

Starting cycletimes are chosen 10 days prior to the dates when TC's were declared ("Start date"). ???? might need to go a little further to allow analysis of pre-storm environment ?????

Tropical storms in the Atlantic Ocean over a similar period,

Tropical storm	First cycle of trial	last cycle of trial	lon/lat when first declared	Category	Comments

Tropical Storm Alberto					On 25 May, 2018 organisation sufficient to classify it as sub-tropical storm; 28 May it became a tropical storm and reached its peak intensity

Tropical cyclones for case study,

Tropical storm	First cycle of trial	last cycle of trial	lon/lat when first declared	Category	Comments

Dorian	20190809	20190909			19 Aug NHC identified a tropical wave in a monsoon trough over western Africa; Cat 5 on 1 Sep

Configuration

As one of the verification measures to be used in this experiment is the ability of the trials to predict TC tracks it is important to resolve the inner core structure of a TC; even better, to have a few gridpoints within the eye of a storm (Q. In order to model the interaction between a vortex and its environment for the purpose of accurately predicting the movement of the vortex do we need to resolve the eye?). To achieve this I would like to run UM at N640 resolution (see here for some issues encountered when changing the resolution of the ported PS41 suite from N320 to N640). However, running at this resolution for a trial lasting for more than 3 months is unaffordable at NCI. So as a compromise I decided to run at N320 UM resolution.

My thoughts on the benefit of higher UM and PFM resolutions,

Higher UM resolution will be able to resolve smaller scales. CX background is then at a higher resolution (more spatial variability). This will lead to more accurate CX columns and hence forward modeling. So the background and observation would be closer. Analysis would fit observations more closely. But on the other hand, CX background will have greater spatial variability and this may move some forward model'ed CX columns away from observations, which will resulting in analysis being away from observations. Which effect dominates?
Improved PFM resolution will lead to improved evolution of increments. With everything being equal this will lead to improved forward model , which leads to .... Result is a better fit to observation.

In short higher UM and PFM resolutions will enable the analysis to better use observations. Q. Is this born out by experiment?

My thoughts on thinning of satellite observations,

The suites have fixed thinning distance regardless of UM and PFM resolutions used. There are 3 rounds of thinning: after checking of preprocess flags (ThinCall 1), before 1DVar (ThinCall 2) and stationlist thinning. At the last stage the thinning distance is usually quite large: ~100 km. This is bigger than satellite footprint size and PFM resolution (for N320).
A new obtype is thinned independently of other obtypes so the new obtype will be used, albeit thinned

OPS

Obstore files were retrieved from the update runs of UKMO operational OS,
- all obstypes are tar'ed in a single tarball
- raijin4:/g/data/dp9/as2291/obstores
- 1.1 GB of data per cycle
- from 20171204 00Z till 20180222 18Z (inclusive)
Ops_CreateODB and Ops_ExtractAndProcess read observations from obstore files
- Details on how to read obstore files and how to be sure all the observations from an obstore are read in correctly: wiki:ticket/370/TicketDetails/OpsReadFromObstore
OPS control/data files
- stationlist
  - thinning distance:
    
    | Obstype | Thinning distance - extratropics | Thinning distance - tropics | | --- | --- | --- | | | | | | AHIClear | no thinning | no thinning | | AIRS | 125 | 154 | | AMSR | 80 | 80 | | ATOVS | 100 | 100 | | ATMS | 125 | 154 | | CrIS | 125 | 154 | | FY3B | 125 | 154 | | GMIhigh | 80 | 80 | | GMIlow | 80 | 80 | | GOESClear | no thinning | no thinning | | GPSRO | no thinning | no thinning | | IASI | 80 | 154 | | MTSAPHIR | no thinning | no thinning | | MWSFY3C | 80 | 80 | | Satwind | 200 | 200 | | Scatwind | 80 | 80 | | SEVIRIClear | no thinning | no thinning | | SSMIS | 100 | 154 |
    
    Q. How does the thinning distance interact with PFM resolution? For each obs Cx is a column interpolated horizontally and in time to the obs location. It is fixed in the inner loop. Cx+ = Cx + S!^-I*(Cw) where Cw is dw interpolated horizontally and in time to the obs location in PFM grid. The observation operator then maps Cx+ to the observation space, y=V(Cx+) to give an estimate of the observed value.

VAR

non-hybrid 4DVar
- alternatively hybrid uncouple 4DVar with error modes from an archive of a single run
PFM resolutions are N108 and N216
- N108 - 185 x 124 km at equator decreasing to 142 x 124 near 40 deg S
- N144 - 140 x 93 km at equator decreasing to 106 x 93 km near 40 deg S
- N216 - 93 x 62 km at equator decreasing to 71 x 62 km near 40 deg S
- N320 - 63 x 42 km at equator decreasing to 48 x 42 km near 40 deg S
used a new VAR build to get around a bug that incorrectly opens RTTOV coefficients - details here

UM

Resolution: N320L70 (0.5625 deg lon and 0.375 deg lat; 63 km x 42 km at equator)
PC2 cloud scheme
????

Results

Overall verification

The trial was evaluated using GES. The GES output shows,

The score cards show the experiment (which used less obstypes and approximates the usage in operational ACCESS-G3) produced slightly degraded forecasts over the control - blue dominates over green - for the first 84 hours of forecasts. But for longer forecast times for the southern hemisphere the opposite is the case. Overall the percentage difference is 0.0 %
The degradation of the experiment (which used less observations) is slightly more pronounced in temperature at 250 HPa in the tropics and in geopotential height at 250 HPa in the Southern Hemisphere
Reduced first-guess departures for the control,
- IASI - more pronounced improvement around MetDB ch 120 which are mid- to lower tropospheric channels (weak, ozone channels), some window channels, around ch 270 (low-peaking, water vapour channels) shows less number of observations used in the experimental run
- similar trend for other IR and MW instruments
The obs count (e.g. this plot with Plot Type=Fit Ratio, Field=Various and Stat=CountOmB) decreased in the experiment
The background fit to observations in the experiment worsened

Note 1. Control (u-bl688) uses all the observations that UKMO use and the experiment (u-bm460) uses those observations that ACCESS-G3 uses

Note 2. In the non-Boite Noire score card plot what's calculated is 100 x (control RMSE - trial RMSE) / control RMSE. Green indicates trial RMSE is smaller than control RMSE. Blue indicates trial RMSE is larger than control RMSE. However in the Boite Noire plots the order of control and trial is reversed: e.g. percentage change in RMS fit to background the calculation uses ( RMS(O-B)_Expt - RMS(O-B)_Ctrl ) / RMS(O-B)_Ctrl

Verification of TC tracks

The GFDL vortex tracker needs following fields,
- essential: MSLP; 10-m winds; 850, 700, 500 hPa u/v winds and geopotential heights (available in the archived files, "*_glm_t???.ff")
- additional: 800, 500, 300 hPa heights
- fields need to be in Grib1 format
- The tracker requires an observed TC location (from either a TC bogus operational analysis file from RSMC Darwin suitebale for the Asian tropics, or alternatively from the NOAA global syndat files obtained from https://ftp.ncep.noaa.gov/data/nccf/com/arch/prod/syndat/) are needed to be used as first guess for the tracker
- Jim to do an initial test of TC tracker once analysis and forecast files are available from a single basetime
  - test whether all STASH fields are output for TC tracker by cold-starting the suite from 20180306T06; long forecast from 20180306T12 should produce forecast files which can be used by the tracker

Refer to here for a summary of the track verification results. (Apologies for the misnamed "G2Tracks" wikipage! Is there a way to rename a trac wiki page?). In short there is very little difference in the TC tracks between the control and the test runs.

Comparison with similar data denial experiment conducted by Fiona Smith

Fiona conducted a similar data-denial experiment using UKMO PS40 baseline global suite. The trial compared the control (u-at590) where only the obtypes that ACCESS-G used at the time of the trial with an experiment (u-ax755) where the full set of obtypes that UKMO was assimilating were used. The trial ran on UKMO's XC40.

The following table compares obtypes used in the control and the experiment,

Obtype	Used in control (u-at590)?	Used in experiment (u-ax755)?

aircraft	Yes	Yes
ahiclear	No	Yes
airs	Yes	Yes
aod	No	Yes
amsr	No	Yes
atovs	Yes	Yes
atms	Yes	Yes
cris	Yes	Yes
fy3b	No	Yes
goesclear	No	Yes
gpsro	Yes	Yes
groundgps	No	Yes
iasi	Yes	Yes
mtsaphir	No	Yes
mviriclear	No	Yes
mwsfy3c	No	Yes
satwind	Yes	Yes
scatwind	Yes	Yes
seviriclear	No	Yes
sonde	Yes	Yes
ssmis	No	Yes
surface	Yes	Yes

Results of evaluation computed on UKMO XC

The results of the trial, TicketDetails/FionaSmithDenialExptPS40 shows that,

A slight improvement in the forecasts from the experiment run which assimilated more obstypes
The improvement is clearer in geopotential height at 250 HPa in the Southern Hemisphere
The forecasts from the experiment is poor for longer forecast times for some fields.

Results of evaluation computed at NCI

The ARD produced from Fiona's trial was used as input to GES which was ported to Gadi. The results show that,

No discernible difference in the forecasts from the 2 runs
Very modest improvement in the experimental run over the control for shorter lead times but worse at longer lead times: e.g. this plot. Overall there is very small improvement in the percentage difference in RMSE
The score card from this trial shows a similar pattern as that from the trial using the PS41 suite: compare u-at590 vs u-ax755 with u-bl668 vs u-bm460. Note the reversal of signs (green and blue patterns) because of the way controls and tests are named.

Comparison with similar experiments conducted by UKMO

The conclusion from the first set of results - that the added satellite observations do not produce better forecasts - is unexpected (and disappointing). To eliminate the possibility that this result is due to an error in the experimental set-up I asked Stuart Newman (SA, UKMO) to send comparable plots. He sent me 2 groups of trials results,

A series of denial experiments where different satellite observations were removed (TicketDetails/UkmoDenialExptPS37)
- There is a consistent degradation of forecasts as different satellite observations were removed
- Relative percentage changes in RMSE is greater than 0 %
- When verifying truth was changed from observations to ECMWF analyses the results are very similar (Jin has plots for these)
SAPHIR impact experiment ([attachment:https://github.com/ACCESS-NRI/accessdev-Trac-archive/tree/main/attachments/wiki/ticket/370/TicketDetails/QJRMS_SAPHIR_MO_Pub.pdf])
- SAPHIR alone produced 1-2 % improvement in mid to upper tropospheric temperature and humidity and a smaller improvement in winds. That result is for the tropics.
- If this result translates directly to my denial trial where not only SAPHIR but a bunch of obstypes were removed then I should be seeing an impact at least that magnitude. But we have zero impact. Why?

Difference in spatial and spectral observation coverage

The null result raises a question: is the assimilation not able to extract the extra information contained in the observing network? Or does the supposedly improved observing network contain no extra information?

To answer this question the spatial observation coverage of the control and the test for a typical cycle is plotted below.

Resources

Jim Fraser's List of TCs in West Pacific & Eastern Indian Ocean from G3 trial periods
- Note "First Lat" and "First Lon" columns show lat/lon when TC's were first named; they are the TC locations on the dates under the column, "Start date"
JTWC bogus central pressure observations are available from Dec, 2017
- the central pressure data are converted to text files and then these text files are read in by OPS

Useful information

Following table has different gridspacing used for various resolutions of UM and PFM

| Resolution | dx x dy at the equator (deg) | dx x dy at the equator (km) | | --- | --- | --- | || | | | | | --- | --- | --- | | N108 | 1.6667 x 1.1111 | 185 x 124 | | N216 | 0.8333 x 0.5556 | 93 x 62 | | N320 | 0.5625 x 0.375 | 63 x 42 | | N640 | 0.3515625 x 0.234375 | 39 x 26 |
The diammeter of the inner core of a typical tropical cyclone is ~300 km (Emanuel, 2018, Meteorological Monographs). So what is resolved for different resolutions is,
- N320 - 5 gridpoints within the inner core, which means inner core is just resolved (using the "4 delta x" criterion)
- N216 - 3 gridpoints within the inner core, which means inner core is not quite resolved
- N108 - 2 gridpoints within the inner core, which means inner core is definitely not resolved

Further work

Repeat the trials at a higher resolution
- Analysing smaller scales may delay the loss of predictability
- What should the ideal background error covariance look like for TC? Will it look like the tropical covariance described by Zagar?

Diary of runs

u-bl688

Cycle time	Failed task	Reason for failure	Action taken

20180224T06 - 20180302T12	Not all batches in MTSAPHIR obstore files were read in; consequently ~1 in 10 observations was written out to varobs	In OPS namelist group, `extractcontrolnl` the variable, `maxbatchessubtype` was set too low	`maxbatchessubtype` changed to main app config value of 40
20180224T06 - 20180304T18	ODB2 files are not archived for this earlier time period	I didn't switch on archiving of ODB	`ARCH_ODB2=True` is used now
20180501T0600Z	gl[mu]_ops_process_background_seviriclear	SEVIRIClear obstore file contains METEOSAT-11 (SatId 70); as VarBC file does not contain parameters for this satellite the task fails	None since it is very nearly the end of the trial period, 20180501T12

u-bm460

Cycle time	Failed task	Reason for failure	Action taken

20180224T06 - 20180304T18	ODB2 files are not archived for this earlier time period	I didn't switch on archiving of ODB	`ARCH_ODB2=True` is used now
20180501T0600Z	gl[mu]_ops_process_background_seviriclear	SEVIRIClear obstore file contains METEOSAT-11 (SatId 70); as VarBC file does not contain parameters for this satellite the task fails	None since it is very nearly the end of the trial period, 20180501T12

u-bm976

Cycle time	Failed task	Reason for failure	Action taken

20190819T1200Z	glu_surf_ascat_ekf	Reading in Bufr file (?)	skipped all SURF tasks; ran glu_um_fcst task without updating soil moisture with glu_smc
20190902T1800Z	gl_ver_hk_ard	ARD_056_008_+7.00000E+02_T00000_G03_u-bm976-GM.con corrupted	Replaced the corrupted ARDfile with an archived copy; note the archive of ARD was done after all tasks of 20190902T00 so forecasts from 06Z, 12Z and 18Z are not in the ARDFile as well as missing verification stats for 06Z, 12Z and 18Z
20190904T1200Z	gl[mu]_ops_process_background_cris	the job failed when updating ODB1	reset to succeeded; not assimilate CrIS for this cycle
20190904T1200Z	glu_ops_process_analysis_cris	Unknown	reset to succeeded
20190908T1200Z	glu_ops_process_analysis_iasi	Unknown	reset to succeeded
20190908T1800Z	gl_ver_hk_ard	ARD_056_008_+2.50000E+02_T20100_G00_u-bm976-GM.con corrupted	Replaced the corrupted ARDfile with an archived copy

u-bn066

Cycle time	Failed task	Reason for failure	Action taken

20190809T06	glu_um_fcst	NaN or Infinities developed	Did a FASTRUN
20190826T0000Z	glu_ops_process_analysis_satwind	Unknown	Set to succeeded
20190826T0600Z	glu_ops_process_background_cris	Unknown; varobs and varcx were created; exactly same error occurred to u-bm976 but it ran after retry	reset to succeeded; ToDo Check if CrIS were used by VAR
20190826T0600Z	glu_ops_process_analysis_cris	Unknown	reset to succeeded
20190827T0600Z	glu_ops_process_analysis_ssmis	Same as what happen to `glu_ops_process_analysis_cris.20190826T0600Z`; TicketDetails/job.out	reset to succeeded
20190828T0000Z	gl[mu]_ops_process_background_cris	failure during updating of ODB1	reset to succeeded
20190828T0000Z	glu_ops_process_analysis_cris	All job.out has `/home/548/jtl548/cylc-run/u-bn066/share/cycle/20190828T0000Z/glu_odb/cris is locked`; because of the failure of glu_ops_process_background_cris while writing back to ODB1 .lock file might have been created	reset to succeeded
20190828T1200Z	gl_ops_ver_odb_obstore_satwind	Looks like error during opening of ODB1	reset to succeeded
20190828T1200Z	glu_ops_process_analysis_satwind	Looks like error during opening of ODB1	reset to succeeded
20190904T1200Z	gl[mu]_ops_process_background_cris	the job failed when updating ODB1	reset to succeeded; not assimilate CrIS for this cycle
20190904T1200Z	glu_ops_process_analysis_cris	Unknown	reset to succeeded
20190905T1200Z	glu_ops_process_analysis_atms	Unknown	reset to succeeded
20190907T1200Z	glu_ops_process_background_cris	Unknown	Run using `OPS_PRODUCEODB=false`

N.B. I made a mistake: starting from 20190904T1200Z the 4DVar tasks of u-bm976 and u-bn066 did not use CrIS

Things to do

What do gl?_ops_process_analysis_* tasks do - generate feedback ODB1? Should these tasks be removed from the suite?
TCBOGUS data seem to be available in UKMO obstores. Will they be used by VAR?
What to do about locally received data?
for gl_ops_process_satwind select based on satellite
for gl_ops_process_gpsro select based on satellite
set up 2 identical twins for a case study: e.g. Sandy, Irma or Dorian
try using the following to fix the problem of job log output files not being retrieved,

[hosts] -> [HOST](/ACCESS-NRI/accessdev-Trac-archive/wiki/HOST) -> retrieve job logs max size
[hosts] -> [HOST](/ACCESS-NRI/accessdev-Trac-archive/wiki/HOST) -> retrieve job logs retry delays

Attachments

QJRMS_SAPHIR_MO_Pub.pdf