Topic 2

• Fully engage in evolving and promoting WIGOS metadata to enable adequate use of data
• Review WMDR2.0 UML model draft
• Fully engage in WIS2 for data exchange
• Position GAW ACDM facilities in WIS and INFCOM/SERCOM

Team B Notes (Judd, Oleg, Tom, Kjetil, Kaisa)

• Fully engage in evolving and promoting WIGOS metadata to enable adequate use of data

  • WOUDC uses for push/pull from OSCAR, pull used for assimilation (planning for push, ops for pull)
  • GALION planning for push/pull from OSCAR, pull for search & discovery tool
  • WDC-RSAT planning for push to OSCAR
  • EBAS planning for push to OSCAR
  • Better for OSCAR to pull from WDC and contributing network data centers
    • avoid problems working with WIGOS schema, and upload API
    • complexity of the representation prevents widespread use and adoption (especially observation element)

• Review WMDR2.0 UML model draft

  • observation element is too extensive, needs minimum requirements for successful upload to OSCAR
    • must be clearly presented and documented to users, with clear examples
    • OSCAR can address issues by minimizing API upload requirements, doesn't affect WMDR2 model

• Fully engage in WIS2 for data exchange

  • comments for WIS2 (NRT data)
  • data centers require brokers (MQTT server). When publish new data, send to broker (link to data). Global WIS2 services subscribe to broker.
    • action item: how-to guide for dummies
  • require WIS2 discovery metadata, same location as data and workflow to broker as the data
    • documentation in progress
    • identify a dataset (data file), metadata is for dataset, not each file
  • registration with WIS2, TBD
  • topic hierarchy, need for atmos comp. Working group membership should include SAG contribution but also us (ET-ACDM) at minimum.
  • global cache for data: retention time is TBD
    • clarify
    • should be NRT only (short time) for contributing network
    • core only, or also some recommended?
    • maintain and share use metrics to data providers

• Position GAW ACDM facilities in WIS2 and INFCOM/SERCOM

  • we are supportive of this
  • promotion of WIS2 to end users is needed

Discussion Topic 2 Team 2 (Atsuya, Alex, Gao, Jörg, Jeanette, Sergi)

For pulling data from OSCAR the WDC need to create an API. Challenges: managing ID for elements you want to upload. WDCGG is working for push mechanism. Ideally WDC push data to OSCAR. Gao changed an erroneous station name and notified to OSCAR about the error. Documentation about the process would be very useful. How to set up the API? who to contact about the errors? GAW suggests sending mails, however doesn't seems the best option. The correction is a problem of the metadata owner. You can create an API where you can modify your own data. Some people confuse WIGOS metadata from OSCAR.

who provides the wis 2.0 metadata? could be linked to OSCAR or WIGOS metadata. WIGOS and WIS2 metadata have different speciation, it can be built in WIGOS the missing points from the WIS MD though. you can attach JSON metadata and add it to WIS 2.0. How consistent WIS MD would be? as it's very simple and flexible, you provide a URL and it queues. To upload in WIS you pack your data and add a limited amount of metadata.

are we going to ask the WDC to fully engage in WIS 2.0? likewise. How are you going to do it? that implies that all the data is going to be push to wis 2.0. It's not persistent, it's only a cache. There could be data gaps, there are not checking. Is WIS 2.0 able to support data uploading as now happens?

There is not guarantee that the client is going to receive all the data depending on the speed of the cache. Global cache for data is important!

Data could be lost. The data in the cache would be still in the WDC. It could be seen as a transport mechanism. Useful for NRT. O3 not submitted once a year but in NRT.

The examples from WIS 2.0 were shown for NRT, but for longer latency it doesn't seem so useful. Calibrations could take time, some data could need time.

We could submit 2 sets of data NRT and fully processed.

Perhaps the mechanism doesn't have value in ICOS but maybe in other contributing networks.

mostly of national stations don't have the capability to upload data like WIS 2.0. Still the WDC would need to do the calibration. The idea is 1 system built for all the WDC. WIS 2.0 could be very beneficial for GAW, for certain data for sure. What happens when the IT instruments fail? GAW needs to be alert about the use of raw (lower quality) data.

In ICOS you receive a notification when you have a persistent identifier, and the data is uploaded, that is not in WIS 2.0. GAW would need a guarantee that the information is received.

To apply I-ADOPT to everything which attributes should be there?

How a contributing network respond to that?

Jörg showed the schematic of the metadata. what is observed?:

• where is the sensor?
• what are you measuring?
• is that measurement representative of the reality?

do the contributing networks need to provide all that information?

not all the camps are mandatory and the WDC can complete the information.

still lack of funding can be an issue.

What would you do if you are using a different standard?

Exercise to compare 2 different systems of metadata: WIGOS vs GEOM.

Summary

Fully engage in evolving and promoting WIGOS metadata to enable adequate use of data

• near term
  • need OSCAR feedback, guidance and focused examples for the data centres
  • need testing and experimentation
• medium term
  • reduce complexity of OSCAR metadata API workflow for publishing to OSCAR

Review WMDR2.0 UML model draft

• need clear examples
  • e.g. cookbooks for stations, satellites
• simplify workflow
• existing standard with additional representations may help (GeoJSON?)

Fully engage in WIS2 for data exchange

• need access metrics from WIS2 global services for data centres
• need topic hierarchy development for atmospheric composition
• need guidance for data centres to integrate and engage with WIS2
  • guidance on MQTT, software options
  • guidance on discovery metadata curation
    • need dataset identification, granularity exercises
  • WIS2 guide will be able to help, need add-on for ACDM

Position GAW ACDM facilities in WIS2 and INFCOM/SERCOM

• supportive
• need to be involved in WIS2, promote/participate

Action items (commitment for the next IP)

• ET-ACDM to work with SC-IMT to develop topic hierarchy
• ET-ACDM to develop implementation plan for WIS2
• ET-ACDM to work with TT-WIGOSMD to develop evolution of WMDR (representation)
• ET-ACDM to work with TT-WIGOSTools to ensure easier metadata management workflow (API)