ET WDC 2020 3 - wmo-im/et-acdm GitHub Wiki

WMO-GAW Expert Team on Data Management

Teleconference via Teams

13:00 -15:00 CEST 11 June 2020

Agenda

  1. Welcome and acceptance proposed Agenda (Jörg 5’)

  2. Approval of minutes from the meeting on 28 April – corrections, comments (Jörg 5’)

  3. Status of DOI Collection, report on issues, path forward (Stoyka 10’)

  4. Progress and outstanding work in harmonization of metadata (Drasko, Tom, Judd, Atsuya, 15' each, total 60’)

    Drasko: "OSCAR DCs Metadata Exchange/Standardisation Status/Efforts";

    Judd: "Lidar Guy Tries to Figure out How to Update WIGOS Metadata";

    Tom: "Python Tools and Workflow for WIGOS Metadata Generation and OSCAR Publishing";

    Atsuya:"About Work to Change_Metadata of the GAWSIS by WDCGG".

  5. Review of Action Items with upcoming/past deadlines - assign person responsible for each task and progress where necessary (Stoyka, all 15’) deferred to next meeting for lack of time ...

  6. Next meeting 14:00-16:00 CEST 7 July 2020 (all, 5’).

Participants:

  • Jörg Klausen (JKl, chair)
  • Markus Fiebig (MF)
  • Tom Kralidis (TK)
  • Atsuya Kinoshita (AK)
  • Kjetil Torseth (KT)
  • Nate James (NJ)?
  • Martine De Mazière (MM)
  • Jeannette Wild (JW)
  • Judd Welton (JW)
  • Debra Kollonige (DK)
  • Gao Chen (GC)
  • Oystein Godoy (OG)
  • Christopher Lehmann (CL)
  • Rayn Stauffer (RS)
  • Stoyka Netcheva (SN, Rapporteur)
  • Drasko Vasiljevic (DV, WMO consultant)

Excused

  • Vincent-Henri Peuch
  • Frank Dentener
  • Enrico Fucile
  • Anatoly Tsvetkov

  1. Welcome and acceptance proposed Agenda. No objections nor additions to proposed Draft Agenda.

  2. Approval of minutes from the meeting on 11 March – corrections, comments. Notes had been available for weeks on group’s website (https://github.com/wmo-cop/et-wdc/wiki/ET-WDC-2020-2 ) but reasons which not clear is not possible to display during the meeting. No corrections reported and this items is left for future reviews.

  3. Status of DOI collection, report on issues, path forward SN: 7 WDCs and Networks provided information on the status of use and plans for DOIs through presentations, comments during the meetings and google doc. Collection of information will be continued. SN proposed to review provided information and if required individually approach contributors for additional information or clarification. Based on collected input and some reference research she will propose a few versions for discussion during future meetings. There was no objections to proposed path forward.

  4. Progress and outstanding work in harmonization of metadata. Summary of activities, findings, repetitive questions, issues, considered approaches, solutions, references.

Drasko: "OSCAR DCs Metadata Exchange/Standardisation Status/Efforts";

Main point of the presentation covered: activities, progress, challenges, resources, considered solutions, approaches and plans for following months. Details were given on MD Structure/Content/Format, Single Station MD Manual Upload method into the OSCAR DB ,Automatic (Short and Long Term) MD Upload method, and XML template. All centres and networks have been engaged -18 of those and progress varies as well as the need of work no matter which method – manual, template or individual XML download, update and upload manually or automatically is selected. Most centers are working on XML updates with different level of prior knowledge, experience and therefore advancement. Test were made with some and XML ready. Lucia from meteoswiss is working with us and helping fix issues or clarify, refine procedures. Documentation is being prepared. It will be easier to do this update in the future.

JKl made a comment that those meetings are indeed to present and discuss findings, issues and vision how to resolve them and share possible solutions and forum to make use of. XML ready means that code lists could be mapped –is it true? Confirmed by Drasko. We are working with NILU and others but we need to have everybody who needs it in order to put any missing elements and to have discrepancies resolved and it to be agree on from us as ToR and variables for composition which also feed to Task Team of WIGOS metadata – codelist update. Vocabularies are sticky points but need to be resolved. To do updates on website everybody needs proper authority based on the role and in order to move on with this work please create ticket to support if you require editing of OSCAR and ask for access.

Judd: "Lidar Guy Tries to Figure out How to Update WIGOS Metadata";

Presented background information, detailed issues, needs of clarity, concerns, gaps encountered during the work trying to align lidar MD with WIGOS in OSCAR. Progress in understanding template and XML, options, plans and information shared which is useful for the community while everybody is occupied with the same task. Detailed issues, errors, need of information and actions are related to templates, process and procedures related to WIGOS, RRR, Code Registry, OSCAR. Proposed method for updating OSCAR using M2M, prepared and shared user friendly template with comments, default and required entries for GAW and additional information as optional entries. Reported and proposed changes: • Provided recommendations and changes in Code registry to better map OSCAR requirements.

• Suggested redefining (and renaming) 5 Lidar Related Variables from Code Registry ObservedMethodAtmosphere as: Backscatter, Raman, HSRL, Ozone,DIAL

• Identified and reported: key lidar related variables missing in OSCAR (and thus code registry); confusing variables; existing variables missing from code registry;

General comments related to remote sensing variables representation in OSCAR were detailed. Provided General comments and recommendations on the existing methods, process to update, upload in OSCAR and align and update MD using M2M process which does not seem to exist. Developed WMDR XML Template for Contributing Networks was explained shared with the group. A list of questions related to variables, process of inclusion and edition, definitions, meaning of code registry and its difference with OSCAR, possibility to expand and make more selective and automated OSCAR search with advantages it could bring and applications it could be used for were presented. It could be very powerful tool for programmatic, regional and global level talks and planning that is important to be updated and this is his personal motivation for doing this work.

Questions, comments:

JKl clarified that station information could be edited only if the person has proper authorisation and it is done at Programme level, National representation level and administration at certain level but not by everybody. Judd’s concern is that each person should be limited to modify information of their own stations only through proper mechanism of control. JKl thanks to the work done and effort made. He has an answer to most questions but time will not permit. The presentation is really useful feedback for the need of better communications and information availability at WMO and OSCAR side of things including RRR link with Code registry. A lot of resources and training videos on OSCAR exists created by WMO and meteoswiss but had been attended my NHMP and not popularized within GAW and now we pay the price. Drasko added – it hard to find them and he had sent the invite to few people on last given seminar. JKl will prepare short presentation answering some questions and from training materials to help the group with some information.

MF shared that he had experienced same/similar problems on vocabularies user semantic terms and inconsistencies. All variables should be use the same of terms homogeneous in variable tables across the disciplines. Assistance need to come from the SAG in good homogenious unified solution. Markus, Judd will put it on github describe issue and others to add to the issue. Gao is dealing with such issues also and will share what he has done and need – size wave, Rh are important. Drasko and Stoyka and JKL to discuss how to move forward –vocabularies important for functional system create section on this issue or other way to deal with this issue.

Tom: "Python Tools and Workflow for WIGOS Metadata Generation and Publishing";

Tom gave an overview of what has been created, applied and available and could be used to generate and publish WIGOS MD in OSCAR. Different methods and interoperability is what we want and it means having variety and loosely coupled interoperability methods. Tom had been involved in schema and XML development, observed machinery and OSCAR development. In the case of WOUDC, WIGOS MD /XML is generated from internal DB and pushed to OSCAR then it is published. WOUDC has its own DB. Station inspectors inspect stations and update info. XML self-describing, easy to exchange info, but it is bulky and has 3:1 ratio content and forms. XML generation is done through pygeometa- small, lightweight and flexible, free and open source tool which supports multiple formats. Configuration is simple YAML through command line or API. It is transformed into WIGOS MD standard- into XML through one tool. Publishing is done by using pyoscar- small, lightweight and flexible, free and open source tool to interact with OSCAR. It could use simple query and upload through command line or API.

Questions, Comments:

JKL in response to question on depl and operational site of OSCAR. Those are 2 parallel environments. As each organization meteoswiss has development, implementation/deployment and operational site. Development site is open externally. Once a week we have mirroring of implementation to operational. Judd is concerned by access of different programs to update MD that are not theirs. OSCAR is open by purpose. Access to change is given to where token is authorised but in order to allo improvement it is given broadly. Updates should be done on XML only where needed so common fields with other program should not be touched is this is agreed on. All changes are logged and changes could be reversed and accesses banned in future if bad things happen. The group should all agree on who and how update on common parts of MD is to be done to have policy, flow in place and mechanism for this. MF: the control/management of OSCAR could help prevent errors in MD. JKl: we need to agree which part of MD NILU and any other programme should provide for updates on stations with multiple programs. Observations? Variables? Instruments? To discuss in future meetings.

Atsuya: "About work to change Metadata of the GAWSIS";

Atsuya presented the steps taken and plan to align WDCGG metadata with WIGOS. He outlined the Metadata and observation data flow and different scenarious encountered by the Data Centre. Metadata fields and data main differences were described which are coming through different data sets and stations and different sets up that need to be harmonized for GAWSIS. Plans and potential approaches to deliver common metadata for the whole center to GAWSIS and OSCAR were presented along with strengths and weaknesses of each approach along with challenges which have been experienced and open to find solution.

Questions, Comments:

JKl: The diagrams incl. MD Venn diagrams are the right approach to the challenges we face now. This is the right approach however we have no time to find answer to them now and have to continue with their discussion in future meetings. We need to agree on data and MD the flow. Information in GAWSIS and OSCAR is changed all the time and stations info is changes, needs and requirements of stations, GAW and DC are other players who and how to change it needs agreement. Please review information and presentations and bring your thought on all MD elements who should provide and change it in our discussions to decide.

  1. Review of Action Items with upcoming/past deadlines - assign person responsible for each task and progress where necessary. This point of the Agenda will be addressed in the next meeting due to lack of time.

  2. Next meeting 14:00-16:00 CEST 7 July 2020. No objections.

Questions, Comments from chatbox:

MF provided information for WDCA & WDCRG DOIs/PDIs status during the meeting. JW: I've not been contacted, but NDACC currently produces XML for download. Let me know what else you need.

AV commented on Judd’s presentation: Great Judd, I was about to make the same simplified XML file.

MF provided the Link to Github issue on variable modifiers: https://github.com/wmo-im/wmds/issues/173 1-01-01 Atmospheric variables · Issue #173 · wmo-im/wmds From discussions with colleagues at NILU: Especially for variables on aerosol particle properties, variable names often need further modification. An example: one such property is the particle ligh...github.com

AV commented on Tom’s presentation: Thanks Tom, looks like an excellent solution to me.

AV added to WIGOS update: What seems to be missing is doing partial metadata updates, is that supported by the OSCAR API?

TK, (ECCC) responded to that comment: I think you can do partial updates by providing smaller XML, in theory. But in pygeometa this would need to be implemented. Feel free to open a feature request at https://github.com/geopython/pygeometa geopython/pygeometa pygeometa is a Python package to generate metadata for geospatial datasets - geopython/pygeometa.github.com

Actions taken in response to issues identified, questions raised, suggestions made:

  1. JKl sent a message to TT-WMD with concerns raised for missing variables, discrepancies between OSCAR requirements and WIGOS code lists, procedures and proposals and referred to github tickets raised. He also requested information on the current status of the WMO activities in this regard and suggest a path of action, specifically who should take the lead and follow-up.
  2. JKl sent a message to TT-WMD requested clarification on how WIGOS Station Identifiers (WSI) can be minted for GAW-related activities. Response provided need to be discussed and further pushed in order to find good path forward which will satisfy the needs of WDC and networks.