ET ACDM 2021 3 - wmo-im/et-acdm GitHub Wiki

Date and Time

25 May 12:00-14:00 UTC (13:00-15:00 Geneva)

Venue

Telecon

Invitees / Participants

  • Jörg Klausen (@joergklausen, chair)
  • Stoyka Netcheva (@Netcheva, WMO)
  • Gao Chen
  • Kjetil Tørseth
  • Debra Kollonige
  • Atsuya Kinoshita
  • Keiichi Sato
  • Markus Fiebig
  • Tom Kralidis
  • Jeannette Wild
  • Christopher Lehmann
  • Dietrich Feist
  • Eduardo Landulfo
  • Øystein Godøy
  • Anatoly V Tsvetkoc

Excused

Agenda

  1. Welcome, acceptance of agenda (5')
  2. Acceptance of minutes of previous meeting (5')
  3. Contribution to GAW Quadrennial Symposium
    1. What are the user requirements? For example, see Difficulties facing data exchange slide 7
    2. Task obtained from SSC
      1. AI-19.7: SSC requests the ET-ACDM to come up with a proposal for unified metadata for all data centers (D: 4Q2021)
      2. Very little in terms of other strategic guidance so far
    3. What are the perspectives for GAW data management for 2025? Role of the WDCs? Role of the contributing DCs? Synergies? Antagonisms? How does this team intend to contribute to the definition of a strategy?
    4. Need a volunteer to write an abstract ... within 2-3 days
  4. Update on TT-WIGOSMD / WG-ACV [jkl 5’ + 10’ discussion]
  5. Status of linking data centres to OSCAR/Surface [all]
  6. AOB [all 15’]

    Participation in the next WMO Bulletin which will be on Data: atmospheric composition data and research perspective.

  7. Next meeting proposed for 23 June 2021 12:00-13:300 UTC (13:00-14:30 Geneva) subsequently changed to 28 September 2021 12:00-13:30 UTC (13:00-14:30 Geneva)

Minutes

1. Welcome, acceptance of agenda

Agenda accepted with no new additions and no revisions.

2. Acceptance of minutes of previous meeting

No comments, no objections. Accepted.

3. Contribution to GAW Quadrennial Symposium

i. What are user requirement? For example, see Difficulties facing data exchange slide 7 ii. Task obtained from SSC a. AI-19.7: SSC requests the ET-ACDM to come up with a proposal for unified metadata for all data centers (D: 4Q2021) b. Very little in terms of other strategic guidance so far iii. What are the perspectives for GAW data management for 2025? Role of the WDCs? Role of the contributing DCs? Synergies? Antagonisms? How does this team intend to contribute to the definition of a strategy? iv. Need a volunteer to write an abstract ... within 2-3 days A discussion followed on submitted and potential Contribution to Quadrennial GAW Symposium

Joerg explained that the Quadrennial GAW symposium is an very important event in GAW cycle as it is a wide consultation with GAW community on setting up priorities for the work of GAW and preparation of its new implementation plan. Data Management and Data Centers are an important component of the Programme discussions are expected to take place in order to determine the direction they should be developed to align with the users requirements. Because guidance was not provided by SSC, ET needs to seek such from the broader GAW community. ET can provide advice on what and how vision could be achieved and construct an implementation plan. The Symposium will have plenary sessions and poster sessions (with break-out rooms) where topics will be discussed – similar to WMO Data Conference. Those two activities during the Symposium aim to engage the community and one poster or talk that will direct the conversation on important for the group’s points will be good for the development of this group in the future. The group still could propose such poster or talk to start a conversation and engage data users and providers within next 2-3 days in order to be included in the programme of the Symposium. What will be those points? What is important for the future of the group? Where we would like to see us as data centers in 2025-2030?

Kjetil shared that he had submitted an abstract related to data licensing.

Joerg asked what will be the concept of integration of remote sensing observations in GAW – can Eduardo and Judd condense an abstract in that context which is based on discussions and considerations made at this meeting with possibility to generalize. What is advantageous, what is strength, weakness, now or historically between remote sensing and in-situ data, how important or feasible is the aspect of NRT provision from those data? According to Gao – the work the members of this group are discussing is having two aspects: creation, modification and alignment of standards and guidelines on one side and Data Management (DM) issues on the other side. DM is more project specific – addresses specific type of measurements and deal with interpretation and the implementation of the standards. Observations from mobile platforms – their role and place, specifics within GAW is another topic. He also informed the group for incoming aircraft campaign in Assia this year. Gao will pass more information on mission planning with the group.

Tom: Between the topics relevant to the work of the team are: unified metadata and interoperability across DC. DCIO group has started work in that direction.

Kjetil: It is what WIGOS is about. Challenges in that directions are existence of individual MD standards which will not be abandoned. This creates issues with vocabularies, standards, solving it is by just creating another metadata standard for exporting the data into for WIGOS/OSCAR.

Joerg: But do we really must keep all MD standards? How those two are working? DCIO concept is to search and find what resources (data) exist for O3 data and find such in NDACC, Brewer network (WOUDC). It facilities data search and discovery.

Tom and Gao: On the other hand, WIGOS is more on stations and instrument side- where the stations are what instruments they are using, what schedule there are using. However, both (WIGOS and DCIO) will be pointing to the same place data are stored/found. So, both are facilitating the discoverability of the data (Data Bases) but looking from different perspective. Granularity of WIGOS is different/high. There is a competition between too much specificity and too wide MD/ description and it is hard to combine. Including temporal and spatial parameters in the search is for higher level of discoverability. It t is tricky to add property search.

Tom: Tier approach could be something that can help: one level search for temporal, spatial, etc then platform etc. search – as one high level search and then parameter search for Data Centers Vocabularies or Code Tables seems to be repeated issue in those approaches.

Joerg: How DCIO relates to WIS, WIGOS and is useful to GAW Data Management?

Tom: DCIO relevance to WIS is the harmonization of MD which is happening with WIS2 and from MD discoverability to WIGOS MD

Joerg: How it will relate to GAW? Will it be one GAW Data Portal (one AC Data Portal) where to search for all AC data in all DC in distributed fashion and is it relevant to GAW at all? Is it start point to further expand to other variables for all our groups in order for them to commit and start moving in one direction? What is the maturity - is it still a dream? Can we start committing even if it is a dream and work and move in that direction? Again vocabulary is different in different centers and also different terms are used in WIGOS historically and practical makes the move difficult.

Gao: By having standards – means high level things and by management – which is specific or deals with how to interpret and best implement this standard in this DB.

Joerg: Is it not what WIGOS is doing?

Markus: If we talk of GAW /WIGOS and DC – they are all specific but different and for DCs it is not possible to implement vocabulary of WIGOS. Mapping to WIGOS can be possible. The working group on vocabularies is merging the gaps and in the future we might have less issues but some issues will remain. 100% map is probably not possible but 50% is too low.

Joerg: Do we think where we will be in 2025 -2030? What is our goal?

Markus and Gao: All data centers will be fully compliant with FAIRENES principles? Vocabulary issues resolved? Mapping will be solved/completed? Achieving interoperability ? Some will be more compliant, others less. We will need qualifiers.

Tom: doing google type search for variable and getting to all providers and linkages to variable names and relevant parameters.

Markus: Typing “albedo” and things like single scattering albedo is suggested … Problem is that those show in different way in different DB - in some linked with underscore in others in other way but finding a way for all to be discovered by mapping all those different concepts. But who will do those links/maps? What role is of DC? DC can provide a developer what needs to be done?

Øystein: what is not clear is how GAW is related to WMO WIGOS and WIS. Those two are important and GAW will link to WMO priorities if those are implemented in GAW. This will straighten the position of Atmospheric Composition in WMO. Such discussions are held in different communities. Also how to work along FAIRNES and anthologies. We have to meet WIGOS and WIS requirements. WMO is not to be isolated from the Research World. Vocabularies server is built but it does not link different variables existing in other communities. It needs to be linked to NDACC community- a link he (Øystein) is eager to see. Links to the academic are indispensable and should be strengthen. It needs to build things outward looking, need the community to develop semantics which is not disconnected from user, inclusive to anthologies, stronger link to the research community. Try to engage them more- in interoperability. An example is Arctic community Polar vocabularies and semantic working group. Good example for move forward is Draft Resolution on Data Policy which is not limited only to core WMO activity.

Dietrich gave an example with TCCON where different organizations try to build one stop shop for all data. GAW and WIGOS are not only ones, Copernicus, GCOS and others exist. They all are good but having different groups puts a lot of burden on data providers, on resources and on making decision on where to go. Lots of smart search engines provide you with search and find data. Important is what it is for the organizations there to join? What is in for them to change to WMO MD standard? Will it lower the barrier for access? Get new information or extended reach of data and expert use, integrate with mass market not geo science limited– both are important?

Joerg: Is the FAIRness the strategic vision for 2030? Are tools for implementation developed? What about QA-QC of data and reports will those be of value for DB and users?

Dietrich: Such things come with price and need justification to fund them. It could work for NHMS but not for everybody. Where data are hosted as well as the user side. International networks are difficult in that regard. They bring not only operational but also political issues. TCCON as archive has been difficult to maintain and would like to integrate with WDCGG or other organization to maintain, or Copernicus Data Store is another option. TCCON was forced to search for a new home with no guarantee for how long it will take to set up and if will not be in the same situation. Hosting centers under NHMS provide stability while might be flexible and provide all services users like to have.

Kjetil emphasized that research data should remain close to /stay with those who know the data. Those Research communities can manage data standards and data accompanying information with overall conventions how to do this – units, and all that is needed. Data now are large and DB should maintain data and information and now required is to build services on top of this. With no scientific lead – this could lead to misinterpretation of data.

Call was made for volunteers to draft abstract for the Symposium. Gao responded that he will try to give his perspective.

4. Update on TT-WIGOSMD / WG-ACV

Concept is to work with isolated issues and develop main metrics for observe properties. Its process is managed by Wiki, including application, streamlined process of review and comments, fixed deadlines, stringent way for constructing with one deadline closing today. TT will meet on Thursday to decide what will be moved forward for amendments. MD model might change upon approval of this process. Under work are aerosols replaced by particle phase; remote sensing and gas phase vocabularies.

5. Status of linking data centres to OSCAR/Surface

Eduardo provided an update on LALINET. The work had been slowed down by COVID which started its 3d wave and hopefully to change in 3d quarter of the year. LALINET held a meeting 3 weeks ago on the whole idea of the Data Center and how to organize itself within GALION. Judd participated and shared experience of what is done in MPLNET to increase connectivity between data centers. This effort depends on the engagement and efforts of the two sides. Tom: linking to WIGOS for ozone and UV is close to completion. Gained experience with OSCAR might be useful for others to find bottlenecks and remove them. Markus: close to implementing DOIs. Progress is made on vocabularies which is a bottleneck. Another thing is also to document provenance. On next meeting will be helpful to inform the group what has been resolved, what is still in the work under vocabulary WG. TICCON activities had been delayed by 6 months now. Include new TICCON software issues to be resolved.

6. AOB

Next WMO Bulletin will be on Data and presents good opportunity to popularize availability, and various uses of atmospheric composition data and to also increase the awareness for the challenges, gaps, and the path WDCs and DB of Contributing Networks are following. The work will involve drafting of text about 500 to 3 000 words with some examples highlighting what the team considers important to show to the community. Examples of past Bulletins and in particular those relevant to GAW are existing and links to them were given along with general instructions for the preparation of the Bulletins. Time-line for having draft ready is mid-August. Call for volunteers to contribute with different pieces for the Bulletin was issued. Individual should contact Stoyka and Joerg in following 2 weeks.

Information Collected in excel spreadsheet on QA-QC tools used by different data centers and contributing networks will be send to everybody at the end of the meeting and allow for updates or to add information that might have been initially omitted. Stoyka will be expecting volunteers to work on WMO Data Bulletin within one-two weeks time.

7. Next meeting: 28 September 2021 12:00-13:30 UTC (13:00-14:30 Geneva)

NEWS

Message from the Chair

Dear colleagues, ET-ACDM members,

Today, 1 April 2021, the draft resolution for a WMO Unified Data Policy for the International Exchange of Earth System Data became available online 0(https://meetings.wmo.int/INFCOM-1-III/_layouts/15/WopiFrame.aspx?sourcedoc=/INFCOM-1-III/English/1.%20DRAFTS%20FOR%20DISCUSSION/INFCOM-1(III)-d05-1-5-WMO-UNIFIED-POLICY-EARTH-SYSTEM-DATA-draft1_en.docx&action=default).

This document is the result of a longer process and a lot of consultations amongst Members and various interest groups. I believe, the Study Group on Data Issues and Policies (SG-DIP) 1(https://community.wmo.int/governance/commission-membership/commission-observation-infrastructures-and-information-systems-infcom/commission-infrastructure-officers/infcom-management-group/study-group-data-issues-and-policies-sg-dip) has managed to produce a draft resolution that is a significant step forward in terms of

  • re-confirming the need for free and unrestricted exchange of data including but going beyond the classical weather and climate observations,

  • acknowledging the importance of atmospheric composition, hydrological and cryosphere, ocean and outer space observations as part of an Earth

System approach

  • acknowledging the requirements, in particular of academia, for attribution and licensing of data

  • addressing private-public partnerships and the commercial weather enterprise

This is a policy document, not a technical regulation, and aspects like the exact nature of licensing (with the intention to guarantee free and open exchange, or with the intention to put certain conditions on use of data) and attribution need to be specified in more technical documents. We hope that this draft is acceptable to and will be supported by Members, and we encourage you to review it. Should you desire to make comments, the mechanism to do so at this stage is through your national delegation to the upcoming INFCOM-1 Part III session, taking place 12-16 April 2(https://meetings.wmo.int/INFCOM-1-III/SitePages/Tentative%20Workplan.aspx?CalendarDate=12/04/2021&CalendarPeriod=week).

Kind regards

Jörg

WIGOS newsletter

The third part of the first Session of the Infrastructure Commission Virtual Session was held 12-16 April 2021.

A side event on WIGOS Tools took place on 14 April 2021.

The Session also adopted the following decisions and recommendations of relevance to WIGOS: • Recommendation on the update of the Guide to WIGOS (WMO-No. 1165), mainly reflecting changes regarding the implementation of WIGOS Station Identifiers in response to Resolution 35 (Congress 18) for the following programmes/systems: Global Cryosphere Watch (GCW), Global Ocean Observing System (GOOS), Global Atmosphere Watch (GAW), and WMO Radar Database (WRD). The issuer of identifier value for observing facilities affiliated with GAW is 20008.

Future Operationalization of GAW and integration with INFCOM At the ninth virtual INFCOM Management Group meeting on 21 April 2021 Greg Carmichael, the chair of the Environmental Pollution and Atmospheric Chemistry Scientific Steering Committee that governs the Global Atmosphere Watch (GAW), presented the infrastructure components of GAW and the connections between GAW and INFCOM were discussed. Some key points of the discussion related to WIGOS are given here.

Challenges regarding the nature of the observational infrastructure in GAW include: (i) stations are not always proposed/registered by Permanent Representatives (PRs), (ii) stations must be registered in the GAW Station Information System (GAWSIS) (required by the Manual on WIGOS) and use GAWSIS to submit applications for affiliation with GAW; (iii) WIGOS Station Identifiers can be assigned by the Secretariat, GAW National Focal points (NFPs) and Network Focal points in addition to PRs; and (iv) interactions between GAW NFPs and OSCAR/Surface NFPs need to be established. Progress was noted; for example, observational capabilities are in general increasing, new instrumentation is being evaluated (low-cost sensors, ozone instruments), and collaboration with the contributing networks is improving. GAW provides extensive contributions to the Rolling Review of Requirements process.

It was noted that under INFCOM there are several elements/centres related to QA and instruments that are responsible for «all components of the Earth System», e.g. the Regional WIGOS Centres, and the Measurement Lead Centres. Their connection with the GAW Central Facilities has to be clarified.

The GAW Expert Team on Data Management works on enhancing data management architectures to facilitate improved metadata exchange and interoperability, data discovery and analysis. GAW data is shared through seven World Data Centres (most are DCPC in WIS) and Data Centres of the contributing networks. Metadata is available through GAWSIS which is a part of OSCAR/Surface. Data have to be submitted to the data centres within 1 year since performance of measurements. Near Real Time data delivery remains a challenge for the research community.

The situation is complex within GAW and there are a lot of interactions with INFCOM including OSCAR/Surface and WIGOS Station Identifiers.

WIGOS training videos

  1. Check out “OSCAR DataBase Introduction” from WMO - OMM on Vimeo. The video is available for your viewing at this link.

  2. Check out “OSCAR Manual Update Using Download Copy Paste And Submit” from WMO - OMM on Vimeo. The video is available for your viewing at this link.

  3. Check out “OSCAR Manual Update Using Edit And Save” from WMO - OMM on Vimeo. The video is available for your viewing at this link.

  4. Check out “OSCAR Web Interactive Tools” from WMO - OMM on Vimeo. The video is available for your viewing at this link.

  5. Check out “OSCAR Web Interactive Tools” from WMO - OMM on Vimeo. The video is available for your viewing at this link.

  6. Check out “Webinar #23 "OSCAR/Surface Release 1.6.0 & Station Templates&quo”;” from WMO - OMM on Vimeo. The video is available for your viewing at this link.

  7. Check out “INFCOM-1(III) Side Event on WIGOS Tools” from WMO - OMM on Vimeo. The video is available for your viewing at this link.

Other recourses

WIS Manuals

WIS Guides

Amendment Processes

WMO Codes Registry

Schema Repository