ET WDC 2020 4 - wmo-im/et-acdm GitHub Wiki

Expert Team on Atmospheric Composition Data Management

Teleconference via Teams

14:00 -16:00 CEST 7 July 2020

Agenda

  1. Welcome and acceptance proposed Agenda (Jörg 5’)
  2. Approval of minutes from the meeting on 11 March and 11 June – corrections, comments (Jörg 5’)
  3. WIGOS Metadata:XML profiles & expected information flow (Jörg, 30’ including discussion)
  4. OSCAR and CDCs MD Exchange/Standardisation Status/Efforts Update (Drasko, 30’ including discussion)
  5. Update on MPLNET WMDR XML Files action item, and relation to GALION and other contributing networks (Judd, 20’)
  6. Status of survey on Data quality control implemented by GAW World Data Centres and Contributing Networks (Claudia, 15’)
  7. Next meeting 14:00-16:00 CEST 1 September 2020 (all, 5’).

Participants:

  • Jörg Klausen (JKl, chair)
  • Markus Fiebig (MF)
  • Tom Kralidis (TK)
  • Atsuya Kinoshita (AK)
  • Jeannette Wild (JtW)
  • Judd Welton (JW)
  • Debra Kollonige (DK)
  • Gao Chen (GC)
  • Christopher Lehmann (CL)
  • Anatoly Tsvetkov (AT)
  • Keiichi Sato (KS)
  • Drasko Vasiljevic (DV, WMO consultant)
  • Claudia Volosciuk (CV, WMO Secretariat)
  • Stoyka Netcheva (SN, WMO Secretariat, Rapporteur)

  1. SN showed proposed Agenda and asked if participants have any additions or suggestions for change. No such were reported.

  2. SN had posted all meeting notes on website where those from previous meetings are and requested participants to review and record need of changes and corrections.

  3. Metadata-flow in GAWSIS-OSCAR/Surface: Interoperability of various (machine) users Jörg Klausen’s presentation included summary of WIGOS standard; WIGOS metadata requirements; 3 OSCAR components: OSCAR Surface; OSCAR Analysis; OSCAR Requirements and one that is not relevant to the group is space based capabilities. Shown was the link to WMO RRR process. Explained was the difference between OSCAR testing and operational environment, links and roles of NHMC, Programmes, WDC and stations, how to register new station or update information on already registered in GAWSIS stations; the difference of GAWSIS program compared to monitoring by NHMS. Given were examples of XML templates with short note on their weaknesses and what flexibility they offer for updates. Each DC could establish template tailored to their needs as some need more information and fields than others. There is no need to create sophisticated schema that is not useful. Template exists for instrument in instr catalog but facility, observation, deployment are based on the same principle. Instrument template is just showing the structure in the presentation which could be found under github. OSCAR surface /station characteristics role is to maintain Environment, programs and central information on observations used by/pointed by any archives. Specific to Observations info such as variables, geometry, instrum methods, data periods, schedules, QA, QC, data policy is PI related and needs to be updated for every archive host by DC or PIs and if agreed on this info flow small risk exists for mixup. Some parts –“extension hook” of that schema facilitates for information that might be specific and informative for some programs but not for others. Presented a slide representing the Draft version of the Role of WIGOS Station Identifier and the differentiation between station registration under NHMS when WIS ID structure is 0-{B}-{C}-{D}, where {B} would be the country code and {D} is Programme specific ID. For GAW Programme Issuer or {B} field under discussion is 21002 versus previously used 20008 (NB: GAW would prefer to stay with 20008). For GAW WSI: 0-21002-{C}-{D}, where {D} is Programme specific station ID/ 3-letter GAW ID and {C} is to be agreed on by ET- ACDM. Under such agreement C could serve the needs of PDIs in terms of unique station ID, and program, and measurement program ID etc.

  4. OSCAR and CDCs MD Exchange/Standardization Status/Efforts Update # 4 Drasko Vasiljevic provided updates on his talk given about 2 months ago. He presented a summary of activities that had taken place, progress and issues that had been encountered while working on project started in February to align MD of different WDC and networks with WIGOS. Several teleconferences and exchanges of information took place within this close to 2 months period with all but 2 centers/networks. Discussion Topics on Telecon(s) and e-mail exchanges included: Structure, Content and Format of metadata (MD); single Station MD Manual Upload into the OSCAR Database; and Automatic (Short and Long Term) MD Upload. 16-17 are XML ready and can produce version on this format. A common problem is the lack of account or understanding what and how to do things incl how to get login credentials and the token and its importance; Discussed were the FAQ OSCAR section; demonstrated downloading and uploading (manually) for the station MD XML. The content in OSCAR is outdated and need updates but also needs tools for further updates. Including easier and simplified way of doing it. Complex process is the main reason behind this outdated information and the difficulty to keep up to date. A lot of things needed explanation and needed are tools to do this updating process– best done behind the scene - with templates, dialog-boxes etc. A number of reference documents and guidelines are under development and others will follow once the processes are clearly understood. Scripts are under preparation to perform simple MD manipulations as search, download, modify and convert csv to XML. Vocabulary mismatch and lack of variables are another encountered issue. Ongoing activities include: edit, modify and upload of information. Big step to faster update will be having more user friendly interface to update DC while MD content is changed.

Questions, comments:

JW Commented to use individual networks instead of GALION

JtW: Concern is that the # station under NDACC in OSCAR 57 is way small than actual when but when she goes and looks at individual stations they all have NDACC association.

JtW To provide list of stations NDACC has but does not show while search done in OSCAR.

JW noticed also something wrong with search interface/ engine of OSCAR- interpretation issue as GALION when the form is interpreted, compound form - if you use Archive is more completed list than the search tool. Downloading NDACC from from Programme gives 76 but in OSCAR is shows 57.

JKl it seems there is serious issue to be fixed. Please provide details- we need your help. The status of NDACC with WMO is also an issue to be resolved.

AT count meaning on slide. GAW stations in WDCR more than 50 stations Is it necessary to correct?

JKl Those stations are not visible in OSCAR to be associated with the centers. Search for variables is supported but not by the data center, it will be tried to be changed if possible.

JtW on XML creation for NDACC. Gao is working on those but under Langley resources. If something is needed now from where data are now residing should be addressed to her and if anything needs to be done she needs to be informed. A lot of exchange exists between NDACC data center and GAWSIS now – do we need to turn it off or replace it or. XML are created everyday.

DV has noticed something strange as what was reported for stations showing or not by Jeannette.

JKl NDACC link is broken now. GAWISIS is part of OSCAR, with the transition to OSCAR those might not be necessary. Drasko should look at XMLs used to be created as this is what we work on now as intermediate format. The xml you produce could be turn in WIGOS compiled DCIO project/standard. In the directory with csv file- cross walk them.

TC we could do as WOUDC - as cross walk medium to long term approach.

JtW Before NOAA data base is turn off, we will reestablish links to external partners. If csv files are no needed is good to know to not further establish in the new database. Relationships with other DB are going to be migrated.

JKl csv could be dropped DCIO xml should be part of the migration plan to NASA Langley. As it could be transformed to WIGOS compliant xml. Send the links on what is done and how to JKl to have a look at those and understand what is the migration process on your end- years to get to the transfer. TC WOUDC is working with NDACC. What and how is maintain active, archive and source for OSCAR information - It is Jeannette who is responsible.

GC try to archive and distributed data in future, first ingest, talking on ingest tools, operation will be different from NOAA, now planning ahead, put automated data collection process in place. We are discussing future move forward. Jeannette is authority on the data.

JKl WMO new structure under Reform created teams (Expert Teams or Task Teams) working on wider areas as metadata steering committee, interaction exists, more work and discussion will follow on needs and on topics such as data and metadata and we can find common grounds and benefit from each other.

From chatbox:

Markus Fiebig WDCA station count: 174; WDCRG station count: 84; Keiichi Sato EANET stations are 57 as of 2018. We will update for OSCAR system; Atsuya Kinoshita WDCGG station count: 206 (fix station: 173, mobile: 33)

  1. Status of survey on Data quality control implemented by GAW World Data Centres and Contributing Networks Claudia Volosciuk gave short update on 9 responses to recently launched survey on implemented Data quality control practices and existing tools among GAW World Data Centres and Contributing Data Centres which takes place upon receiving data at WMO Data Centre/Network data centers and networks. She shared the link to the survey on chatbox. It takes 10 -25 minutes to complete. Results could be presented at next meeting and Claudia is open to answer any questions that exists. Claudia will follow up one by one and encourage everybody to do this. KS, MF committed to submitting information at the meeting. DK and JW ready to provide additional information if needed.

  2. Update on MPLNET WMDR XML Files action item, and relation to GALION and other contributing networks Judd Welton: Update on Action Item updating entries on his networks not done for 10 years. Now have ability to extract WIGOS XML files and search results (json) from OSCAR. Will be able to update XML files directly from MPLNET database & upload to OSCAR or populate a template and upload that to OSCAR. It will be automated in future (update daily or monthly). Results/files are stored locally. Library routines were developed to read/manipulate/save. Next step is testing uploads back to OSCAR API. Added WIGOS parameters to internal MPLNET DB such as station IDs, type, geometry and prepopulate instrument level information etc. Approach is first updating internal DB then uploading in OSCAR. Went through OSCAR archives which show 73 MPLNET sites and only 9 are affiliated to the Programme in OSCAR and 29 are matching in OSCAR using 1km criteria. 20 sites to be affiliated and 44 to register and affiliate. Continue to have issues with variables – missing or in requirements but not in code list. Major road-blocks are OSCAR accounts and permissions (for tokens to fully utilize the API) and still waiting for a token. Issues and progress on work with WIGOS XML files: proposed template approach will facilitate contributing networks update and upload via API. Problems exists when extracting info using API- missing elements from data structure files – is there required list of elements? Looks like to standard structure exists. If missing /omitted this complicates programming reader apps. Proposed solution to have fix API file content and fill missing elements with NULL. Lack of wavelength information for remote sensing.

Questions, Comments:

JKl thank you for extremely useful information such as reported deficiencies in standard, OSCAR implementation, API, etc. Lots can be fixed may be not in short time but we will try as many as possible of those to be resolved. Please communicate findings to search for solution. Israel station Sede Boker is an example where ET can productively interact to solve problems in quality in MD (naming, coordinates, etc). PI of the station is best person to know, but in this group we can identify issues.