ticket_193_TicketSummary - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki
Updating agrepsGP to use OPS30.0.0
Preliminary
-
GE1 used OPS27.2 and the obsgroups used were SURFACE, SCATWIND, AIRCRAFT/SONDE, SATWIND, ATOVS, AIRS, IASI (separate smsfile), GPSRO.
-
GE2 uses OPS30.0.0 and additional obsgroups are introduced: SSMIS, CrIS, ATMS, MTSATCSR
-
In this work following OPSUI jobs are used to update agrepsGP suite tasks:
| ID | Owner | Obsgroup | Remark |
|---|---|---|---|
| oamkn | vinodk | SURFACE | |
| oajsm | pgregory | SCATWIND | |
| oamko | vinodk | AIRCRAFT/SONDE (???? separate jobs ????) | |
| oajsl | pgregory | SATWIND | |
| oamkb | vinodk | ATOVS | |
| oameb | fbaordo | AIRS | |
| oamec | fbaordo | IASI | |
| oamkl | vinodk | GPSRO | |
| oamka | vinodk | SSMIS | new in APS2 |
| oamge | jinlee | CrIS | new in APS2 |
| oameh | fbaordo | ATMS | new in APS2 |
| oameg | fbaordo | MTSATCSR | new in APS2 |
Modifying suite tasks
4 scripts need modifying to allow OPS30.0.0 to run within the SMS suite. This is the procedure that we will follow:
Following changes were made to "OpsList_Main" to run OPS30.0.0
| Parts of script needing modification | Actual modification |
|---|---|
| OPS_DOMAINFROMHEADER | changed from false to blank (i.e. no value) |
| OPS_MAKE_NEW_ODB | changed from true to false |
| Fortran namelist groups, &GeneralControlNL | remove all namelist variables except "CapSupersat=F" as default values given to these variables are correct |
| ???? VarobsVersionNumber may need to be set depending on how old LETKF is - it may not be able to read varobs which has VarobsVersionNumber=4 ???? | |
| Fortran namelist groups, &QcNL | remove this namelist group entirely |
| Fortran namelist groups, &ExtractControlNL | &ExtractControlNL for IASI, SEVIRIClear, GOESClear are not needed |
| Fortran namelist groups, &FieldsControlNL | retain just 'ForecastPeriods=9,12,15' |
| Fortran namelist groups, &ProcessControlNL | &ProcessControlNL for IASI, SEVIRIClear, GOESClear are not needed |
| Fortran namelist groups, &VarobsControlNL | not needed as default set of VarFields should be O.K. |
| Fortran namelist groups, &CxControlNL | ???? should some be removed ???? |
Following changes were made to "OpsList_IASI" to run OPS30.0.0
| Parts of script needing modification | Actual modification |
|---|---|
| OPS env vars, export OPS_GPSROCOEFFS_DIR | removed |
| OPS_DOMAINFROMHEADER | changed from false to blank (i.e. no value) |
| OPS_SSMIS | changed from true to false |
| OPS_SEVIRICLR | changed from true to false |
| OPS_GOESCLR | changed from true to false |
| OPS_MAKE_NEW_ODB | changed from true to false |
| Fortran namelist groups, &GeneralControlNL | remove namelist variables except "CapSupersat=F" as default values given to these variables are correct |
| ???? VarobsVersionNumber may need to be set depending on how old LETKF is - it may not be able to read varobs which has VarobsVersionNumber=4 ???? | |
| Fortran namelist groups, &QcNL | remove this namelist group entirely |
| Fortran namelist groups, &ExtractControlNL | &ExtractControlNL for all obsgroups removed except for IASI |
| Fortran namelist groups, IASI.nl -> &ExtractControlNL | 1. namelist variable, MetdbSubtype changed from "IASI" to "IASIG" |
- Added BufferSize=2500 and MaxBatchesSubType=40 (are these needed for extraction from obstore????) | | Fortran namelist groups, &FieldsControlNL | retain just 'ForecastPeriods=9,12,15' | | Fortran namelist groups, &ProcessControlNL | &ProcessControlNL for for all obsgroups removed except for IASI | | Fortran namelist groups, &VarobsControlNL | not needed as default set of VarFields should be O.K. | | Fortran namelist groups, &CxControlNL | not needed |
Following changes were made to "EGP_ProduceModelObs_Main.sms" to run OPS30.0.0
| Parts of script needing modification | Actual modification |
|---|---|
| OPS_BACKERRUM | removed as this is used only in BackErrCreate - looks like a confusion |
| OPS_UMBACK_LIST | This env var should point to CX background so it now directly assigned to $PP_012_DIR/ens$ENS_MEMBER.pp_012.$yyyymmddhh_m12h |
| Stationlist | ???? Need to use same stationlist as deterministic ???? |
| Bias regression coefficients | ???? What coefficients need to be used - for each obsgroup OPS runs N times using at each time CX background from one ensemble member; so in theory the bias coefficient should be unique to each ensemble member ???? |
Following changes were made to "EGP_ProduceModelObs_IASI.sms" to run OPS30.0.0
| Parts of script needing modification | Actual modification |
|---|---|
| OPS_BACKERRUM | removed as this is used only in BackErrCreate - looks like a confusion |
| OPS_UMBACK_LIST | This env var should point to CX background so it now directly assigned to $PP_012_DIR/ens$ENS_MEMBER.pp_012.$yyyymmddhh_m12h |
| Stationlist | ???? Need to use same stationlist as deterministic ???? |
| Bias regression coefficients | ???? What coefficients need to be used - for each obsgroup OPS runs N times using at each time CX background from one ensemble member; so in theory the bias coefficient should be unique to each ensemble member ???? |
Comparing ensemble OPS tasks
For each obtype one varobs file from the old ensemble OPS task for one member is compared with corresponding varobs file from the newer ensemble OPS task. The varobs files compared are:
ngamai03:/g/sc/home/jinlee/da/ens/data/test_ens_ops/old/Process04.varobs
- Copy of ngamai03:/g/sc/fs4/flush/dhsmith/GP1/2015051800/etkf/EnsembleOPS/Process04.varobs
ngamai03:/g/sc/home/jinlee/da/ens/data/test_ens_ops/new/Process04.varobs
- copy of ngamai03:/g/sc/fs4/flush/dhsmith/agrepsGP2/2015051800/etkf/EnsembleOPS/Process04.varobs
| obtype | total # obs in old varobs | total # obs in new varobs | remark |
|---|---|---|---|
| Aircraft | 25795 | 24844 | as far as matching obs are concerned they are identical |
| Sonde | 976 | 973 | For matching obs obs values are slightly different; PGE's are different too |
| Surface | 38019 | 35594 | For matching obs obs values are; PGE's are different |
| AIRS | 2315 | 7182 | print-varobs produces large files |
Notes
- "OpsList_Main" processes all obsgroups except IASI. This is different to the way OPS tasks run in an assimilation cycle where one OPS task processes one obsgroup
ToDo
| Task | Description | Status |
|---|---|---|
| All control, coeffs, bias files need to point to BNOC directory. Also the version numbers should be checked against those in UKMO collab wiki page | open | |
| varobsversion (????) should be checked so that subsequent jobs - LETKF (?) - can read varobs | open | |
| Is the format of CX background right as an input to OPS30.0.0? | open | |
| Is the format of bgerr right as input/output to/from OPS30.0.0? | open | |
| Check TrimObstore accepts new obsgroups | The inputs to TrimObstore are obstore and varobs files for different obtypes. The obstore files contain exactly the observations contained in ODB's which were used by the deterministic OPS. The varobs files were also produced by the deterministic OPS but they contain subsets of observations after a series of quality control steps and thinning. TrimObstore produces output obstore files which contain the same observations as corresponding varobs files. These trimmed obstore files are used to update inflation factor (how?) | open |
| On p3 "ETKF Documentation" there is a remark about $ENS_PERTVAROBS_DIR and $ENS_CURVAROBS_DIR. According to this document deterministic varobs should no longer be used by TrimObstore and instead control varobs be used. Also see p6 of the document | ???? | ???? |
| The ETKF transform matrix calculation is done at N216 resolution - same resolution as the forecast part. In this upgrade need to check any parts which may depend on resolution | The parts which may be resolution-dependent are: |
- CX background | Open | | An inflation factor is multiplied to the raw transform matrix for each localisation centre. Each inflation factor is made equal to the ratio of the rms error of the ensemble mean (i.e. innovation vector = y_o - H(x_bar), where x_bar is the mean of forecast ensemble from previous cycle and y_o is the vector of current observations) and the rms spread of forecast ensemble (in observation space). This ensemble spread calibration also requires accurate observation error variance (R). For this reason only certain observation error variance and hence certain observations are used. Observation groups which are supported by ETKF can be found in EnsMod_Varobs/EnsMod_VarobsNumber.f90. It looks like current ENS source code doesn't support SSMIS, CrIS, ATMS and MTSATCSR | | open | | Compare varobs/modelobs files from OPS tasks of the control SMS suite with those from OPS30.0.0 | Use print-varobs and other OPS utilities | ???? | | Need to decide on setting OPS stationlist, bias coefficients files, SatRad coefficients files, MetDB elements files | ???? | ???? | | ???? | ???? | ???? |