ET WDC 2020 6 - wmo-im/et-acdm GitHub Wiki

Date and Time

16 October 2020, 12:00-13:30 UTC

Venue

Telecon

Participants

  • Jörg Klausen (chair)
  • Stoyka Netcheva (WMO)
  • Martin Schultz
  • Tom Kralidis
  • Alex Vermeulen
  • Lorenzo Labrador (WMO)
  • Judd Welton
  • Markus Fiebig
  • Jeannette Wild
  • Eduardo Landulfo
  • Dietrich Feist
  • Gao Chen
  • Lucia Cappelletti
  • Kjetil Tørseth
  • Debra Kollonige
  • Ryan Stauffer
  • Atsuya Kinoshita

Excused

  • Drasko Vasiljevic (WMO)

Agenda

  1. Welcome, acceptance of agenda (5')
  2. Acceptance of minutes of previous meeting (5')
  3. ET-ACDM at 2020 WMO Data Conference
  4. Status of metadata delivery of archives to GAWSIS-OSCAR/Surface (DV, 30')
  5. Brief on joint-SAG meeting (SN/LL, 10')
  6. Brief on KoM of INFCOM / SC-IMT / ET-Metadata (TK, 10')
  7. Recent activities of WDCs / Contributing networks archives (30')
  8. Next meeting
  • 10 December 2020, 13-15 UTC

Minutes

JK: Short brief on WMO Data Conference from Joerg Klausen on request by Martin Shultz. JK contributed to preparation of Res 42 – to deal with data policy. It is high level activity with WMO DC with governance questions to be addressed. JK presenting GAW data policy incl everything presented in February for the study group. If anybody wants to send input will be included. ET input will be solicited prior to the actual conference and properly represent the community. My message to convey our community is heterogeneous, outside of NW Services, to large degree science driven with some observation operational capacity and activities. Data policy is mostly open and for FAIR use. Data policy is one thing but licensees and rights of what do or not is very important aspect. Those are not currently aligned with res 42 in the draft version now. Send input which JK will include.

Please provide brief updates on where you are in the process of updating metadata. I invited Lucia at today's meeting because she is responsible for the operation of OSCAR from Meteoswiss and she is also the most knowledgeable person on OSCAR, can answer questions, can tell us on any features intended or added.

SHADOZ update by DK: All 14 stations in DB and 2 stations did not have affiliation with SHADOZ which we corrected manually. Data are in DB. General information typical things – instrument and methods do not change much with time. Questions sent to Drasko related to metadata. One of the challenges is related to “mandatory” versus “optional” information. A lot of stations had been established with other observations when Shadows measurements started at those stations. When editing metadata information how to organize shared editing in MD files – question which Judd also mentioned before. Is there process of shared editing of the files? Several DC have data sent in NRT - how often such information MD for station to be updated ? For SHADOZ – data are not NRT and we often do quarterly updates with data sets – do we need do update MD quarterly or when MD change? To answer questions at the end.

TK on WOUDC – almost there with WIGOS MD updates in oscar. Managed to find suitable way and extract information now to formulate xml the way we did for NWS with few changes. 3-4 weeks we will have some updates and then every weeks we can keep Up.

MF on WDCRG: have xml profile filling in what we can updates. Bottle neck is we have many variables which are not in OSCAR DB. We have a couple of open issues in github which are dormant since the summer. So first we need to make progress on those issues in github in order to advance our collection update in OSCAR.

JK: Complexity of changing the structure in terms of MD model, then application and then model and takes fair amount of time due to downstream dependencies. Provide what you have as existing variables and then we could work together as adhoc teams as atm comp experts to find solution of vocabulary problem. You were to set up of telcom with me and few other experts.

MF: I propose interim solution to make progress without having this huge process as a bottleneck and if we agree now then we can submit vocabulary without those modifiers and make progress and make the update to WIGOS metadata data record while we work on those missing.

JK: Yes, absolutely the TT is not started. ET started but not underlining TT. WMO become more rigid the way codes have to be managed and that is why things had not been moving. Other solution to create and use existing variables following non ideal approach that WOGOS MD has in place now. MF: if we do this we will be probably locking ourselves in position that we want to resolve

JK: not exactly but providing workaround. I can not seed up this process. Any change is to be in June next year- formal process of WMO and beyond of my control.

MF: what I propose is to agree to for all var to define use approach leave open how to solve variables when we resolve this modifier issues.

JK: we could achieve this through the work group I proposed to form we need telecon and TT outside of this groups. Please suggest time to discus and converge our ideas and how we can proceed and consult other people. It is time consuming and difficult process we can progress if we start. EBAS – is big contributor for GAW – for any technicality (that has nothing to do with vocabularies) preventing to producing and link information in operational manner be resolved so et least a subset to be provided. Not sure where Drasko went with mediating this process. Sugest day and time for JK and Lucia and we will figure it out.

MF: Yes. But I still have a question I suggested but lets discuss outside or this call.

AK: WDCGG: Working is pending. Division Staff are relocating to another building and busy with this. Using WOUDC xml to update the metadata but had problems. Expected that OSCAR update in September to change metadata but it did not happen.

JK :update of OSCAR will not change metadata. It will not be done. Source of metadata is WDCGG and OSCAR can not change it. Give us clear guidance what you need to be fixed. And AUIP to help- changes. We normally do not change the metadata. Use OSCAR feedback form for this. Or e-mail us.

Lucia Cappelletti: suspecting the new link was providing via xml but instead of replacing the new link was added. How did do provide new url –with xml or on the user interface?

JK: send your presentation to Stoyka and JK to put on github as reference and communicate to OSCAR teal directly – Lucia and use support form to understand the problem and solution you propose and solve the problem.

Dietrih: TCCON. We started in May. I was really aware of this OSCAR and those activities. We are currently reprocessing all data set as we have new retrieval software and will be adding the metadata. We have DOIs for all download files with lots of MD data about 80% from what you want to see in OSCAR. We want to do this for upcoming data reprocessing to finish end of December and early 2021. XML generation is realistic to be done in the Spring.

AV: ICOS started in June. Looked at GAWSIS system. Recently we become Contr. Network, we started manually updating metadata in GAWSIS and were added as contributing NC with 35 stations. We are reflecting this change in GAWSIS for all stations. Now we are working on updating all information, instruments, stations, not only on surface but also ocean and ecosystem stations and it is a lot and challenging. It will take time. We have people. We continue as fast as we can. Changes: comments from WMO National Reps complaining that we are updating information but they did not like it even for my own stations. Another challenge: in GAWSIS it is not clear at the time at which point what information you are editing at the moment. Working on this update on MD, working to synch our metadata with that in WIGOS.

JK: you can add to wiki this issue to see and discuss solution.

AV: Issues are coming because I informed PIs of the stations as some stations had not been updated for long time. Complain came from national representatives with (GAW) WMO. NReps wants us to get permission to change information in the system.

JK: Automated process is set up to improve timeliness and accuracy in the information system. WMO has to resolve it. OSCAR – goes beyond Temperature and Pressure etc. and covers variables of composition which are not with NHMS. WMO NPReps feel they are bypassed. This is difficult governance problem to be resolved and can be at high WMO level- rights and who can deal with such information. Need to see how to make it as an action item as it will affect all metadata updates.

AV: as our information is not going trough NHMS but other institutions it causes problems. Those stations have GAW ID. I informed station PIs of changes

SN: everybody who has similar issues to send information to SN in order to be passed to WMO management to work on solution.

JW: MPLNET not done anything since last update(s) as I spoke to multiple meetings. I reported numerous issues and problems which are now being discovered by others. Some are similar to the variable issues mentioned today. I am on hold waiting to get resolution to what was reported in github. I encountered a couple xml uploading and DB on our end many were resolved except day time in observation element issue and I am waiting on resolution of that one also waiting for some information on the update of OSCAR –GML ID and its expansion as it will alter the code I have that generates xml. I need information on those.

JK: Who do you expect info from?

JW: From whoever can give me such information – something has change in GMLs but no description how exists on what. Somebody needs to compile and provide such information and provide it. I hear we all seems to be going through experiencing similar problems on generating XML files and manual edits. Need clear description on what and how to do it so not everybody is going through the same difficulties as I did. Tom is similar situation working on observation element of information and encounters issues. I need information on new GML schemes mainly. I suggest separate objectives here and have people working on updating metadata in development environment not on the operational side because of issues with sites and the stations and the governance otherwise it will never work. Updating metadata can be worked on development environment not operational and at that one it will not trigger issues with national representatives. Because to figure out what and how to be done takes a lot of time and can be done in development environment. All testing and development should be done not on operational side.

Lucia Cappelletti: New version and what changed and where the info could be found. End of Sept user manual was updated also last week. Infor on how it works and GMLs, table contents used for matching. This document is good start not the end of the story. Business logic is complex and different and difficult to describe each case. URL will be posted on chat and it is in OSCAR. Users to be alerted for this is good but hard to figure out who needs and what this information. Remark is put for the new release. How can we better inform people is welcome input from you. Use this opportunity. Will put the link to user manual in the minutes.

OSCAR user manual 2019

Eduardo Landulfo: LALINET with about 13 stations: get advice from JW on how to make xml, what kind of software he is using and how to work with them and he kindly provided those me as he will apply them latter to other lidar networks. We are updating information internally and DOIs for each station are applied. We received direction from Drasko and Lucia how to deal with OSCAR to upload xml and want to thank you Lucia. The situation – many stations are with university and closed in this time and no personal to deals with them and takes longer than we want to.

JK: Thank you for your efforts, than you to Judd for helping and Drasko and Lucia.

JW: is this Surface user manual the final version.

Lucia – yes it is final. Due to approval process it will be published actually in months.

LL: provided a brief on recent joint 3 SAGs meeting SAG RG, SAG Aerosols and SAG on atmospheric deposition in September 2020 in 3 days 2 hours each day. To align different SAGs common activities and links and works with Expert Teams and also to respond to requirements and direction from WMO Reform. Was done remotely dues to COVID situation. Chairs of Experts Teams were also invited. To benefit from joining resources – RRR, QA-QC, WDC and flow of information to improve as well as input and engagement. We are advancing understanding of WIGOS and OSCAR. This is just the start.

The meeting

• Conceived by the Chairs of the Aerosols, Reactive Gases, and Total Atmospheric Deposition SAGs

• Purpose: explore common scientific challenges and future avenues of collaboration among the Aerosols, Reactive Gases, and Total Atmospheric Deposition SAGs within the framework of the new WMO structure, and lay the groundwork for a working relationship with GAWs' new Expert Teams and Science-for Services Initiatives (MMF-GTAD, GAFIS, IG3IS).

• Chairs of ETs and MMF-GTAD also in attendance

• Originally a physical meeting

• First of series of meetings

Outcomes

• Informing the SAGs’ membership of new collaborative framework in the three SAGS

• Shed some light on the role of ETs and their interaction with the SAGs  RRRs, essential variables, data accessibility, GAWSIS-OSCAR, contributing networks, SAGSET information exchange

• Links: MMF-GTAD is now part of the three SAGs’ agendas.

• First discussions between the SAGs on common scientific agenda.

Next steps

• Firm up the three SAGs’ common scientific agenda

• Define the SAGs’ common data, data management issues related to the ETs and address them jointly when possible

• Individual SAG meetings for post-mortem analysis

• Early days and hopefully the first of such meetings

Reactive Gases SAG meeting took place yesterday – GAWSIS OSCAR was on the Agenda yesterday. JK gave presentation – emphasized SAGs input is very much required to redefine or evolving vocabularies.

TK: Brief on kick off meeting of INFCOM Steering Committee on Information Management Expert Team on Metadata. One of INFCOM committees. Inside many ETs and TTs. Topics on Data or Information System. One is ET on Metadata Chair is TK and JK is vice chair. ET is tasked with alight metadata in WMO with associated MD in other standard and organizations. Tasked with development Documentation, Guidelines and Best practices. Identifying tools for WIS or WIGOS MD for WMO. How better enable WMO organization to produce MD. MD to meet one standard schematically is not enough need semantic to be more clear, to have assessment framework with KPI is big piece in the puzzle. TT – list of those for WIS… WMO information system is moving towards version 2 implementation which modernized technology, workflows etc with WIS architecture where MD is essential. Developing code list manual, standards etc. There are cross cutting issues to support feedback. How to deals with evolution of standards, changes in OSCAR. Capability, applications and impact on them. Change management. High level workplan exists. Now membership is clear across agencies. Kick off of TTs, Code list, KPI, transitioning to WIS 2. Can circulate the presentation to the group.

Expert Team on Metadata Standards

Standing Committee on Information Management and Technology (SC-IMT)

MF: What is the relation between WIS MD and WIGOS MD and what are used cases or Application area?

TK: This is soft market wonderbased (?). One of cross cutting discussion groups if we can have single MD format, discovery opposed to observation, instrumentation MD – now WIS is more google MD with reference to WIGOS MD. We want to move forward to one united MD standard in long term.

JK – WIGOS is more oriented to describe observations, WIS to products, and in some case same some is not. In my mind we can converged to common and have profiles to be used in those used cases. Formal restrictions of WMO to engaged external experts if not nominated in National DB. Engaging domain experts is essential and must engage external experts all WMO activity will fall out and can not be productive. JK had many discussions with WMO management. Community must be engaged even if they are not formal members of ETs or TTs.

[10/16/20 3:27 PM] Welton, Ellsworth to TK - Can we see a list of the members of these metadata task teams? Or alternatively get this presentation copy?

AI1: MF to set up a meeting with JK on discussion way around for missing variables now and path forward with new TT.

AI2: JW to confirm if his question on GML is in new OSCAR manual and if not to contact OSCAR or Lucia.

AI3: Information on issues coming from WMO PR on OSCAR station updated to be provided to WMO Management to find solution (JK, SN, rest of the team, OT)