W2IT Priorities (23 June 2025) - wmo-im/et-w2it GitHub Wiki

A/ Global Cache data volume

[Jeremy] Data volume growth in the (UK/USA) Global Cache. The Tech Regs suggest 100 GB volume. But as of May-2025, we're on > 500 GB. Here are some statistics for how much data is coming from each WIS2 Node ... Here are some statistics for how much data is coming from each WIS2 Node ...

Picture1
We have 7 centres shipping more than 500 MB (I didn't include the rest).
Canada is by WAAAAAY the biggest at 416 GB. Interesting to see that it's 10x the size of the GTS feed published by Germany! ca-eccc-msc sets cache=false for NWP and model data already.
Tech Regs allow for the possibility for the GC not to cache data that should be cached (core and cache: true or nothing) and inform the originating centre about this. Currently the UK/USA GC doesn't have this functionality implemented yet.
I think not caching stuff on "experimental" topics is fair game. They shouldn't be used for operational purposes anyway?
Not caching anything in "prediction" feels a little blunt to me.
Maybe a blacklist of topics (with wildcards) would be better? So we could filter out from a particular WIS2 Node, and maybe be a bit more specific on which sub-type of data is a volume problem. e.g.,
origin/a/wis2/ca-eccc-msc/data/core/weather/prediction/forecast/+/probabilistic/ +
This would catch all the ensemble data irrespective of forecast type.
Thinking out loud on metrics, it would be useful to generate another metric for the total volume of data cached for a given centre-id. With prometheus we could easily work out the change over previous 24-hours. E.g. "wmo-wis2-gc-cached-volume-total", labels "centre_id|report_by", description: "cumulative volume (in bytes) of data cached from a given centre-id". This might me an interesting metric to see from all Global Caches and would also allow a quick comparison to see if there are big differences in volume from one GC to another? +
A GC should capture metrics about messages/topic/centre-id, to be able to sort descending, to trigger communication to a WIS2 Node (monitor/a/wis2/...). This metric can then be used by a GC to determine whether or not to discard voluminous data that is not properties.cache=false bound (and emit a message via monitor/a/wis2/...).
Values could be reported back to WIS2 Nodes via monitor/a/wis2/$GLOBAL_CACHE/$WIS2NODE

${\color{green}{B/ \ Implementation \ of \ Global \ Replay \ service(s)}}$

[Remy]approve the specification Who to implement? Step1: endorse the spec (PR:htps://github Step2:official SG request Members to provide this GS Step3: Canada and South Korea offer to run Step 4: encourage Members to use the G replay Service- particularly ECMWF and other NWP centres Canada can do this with an official request from SG? Remy, normally the request comes from Members See also: https://github.com/wmo-im/wis2-grep

${\color{orange}C/ WIS2 \ Node \ backup \ possibility}$

[Remy] Add backup option for WIS2 Node:

As you know, at the moment, with the agreed specifications for the WIS2 Node, and for a given centre-id, it is not possible to define more that one MQTT address for the local broker of the WIS2 Node. It means that all Global Brokers are subscribing to this unique mqtt(s) address. If, for whatever reason, this mqtt(s) address is not reachable any more, then the WIS2 Node is "disconnected" from WIS 2.0. We even have an alert for that ! In the FAQ that is currently being developed, we have the question of this aspect of "backup" WIS2 Node. I have been in contact with our colleague from Belize who is really interested by this feature. I also had exchanges on this subject with Poland when they implemented their WIS2 Node. Do we want to define a standard backup option ? If yes, how to do it ? Usage of the DNS and switching from one IP address to another looks tempting. However, DNS has not been designed for that and, from my experience, it doesn't work at all or at best badly. Considering the nature of MQTT, the Global Broker always have a connection to the local broker of a WIS2 Node. It means that it is possible to detect the loss of this connection and somehow (NOT using the DNS !!) switch to a secondary mqtt(s) endpoint. And if the connection to the first broker is reestablished, it is possible to revert back to it. A NC/DCPC willing to use this option would need to operate two brokers and very likely two webservers. They would have to manage both of them, providing data to both, notification message on each broker... Global Brokers would have to ensure that only one of the two is effectively "active". in an upcoming ET-W2IT meeting, and with the agreement of the Chair, I would like to table this discussion, whether having a standard way to improve the overall redundancy level of the WIS2 Node is useful. If yes, how to do it...

GitHub Issue #2 Add backup option for WIS2 Node · Issue #2 · wmo-im/wis2-implementation-transition Possibly draft a cookbook section on WIS2 Node backup

D/ Change management for Global Services ${\color{red}{(relevant\ to \ ET-WISOP)}}$

[Jeremy] Notifying stakeholders of changes etc. For example, GDC OGC-API Records end-points from Canada and Germany functionality has been unstable over the last month (as tested by EUMETNET RODEO/MeteoGate).

E/ Clear Guidance needed on whether data should be marked as Core or Recommended

[Jeremy] Core vs Recommended decision shouldn’t be conflated with the open licensing

License Type
Data Policy Open Restricted
Aligné à gauche Core Yes No
Aligné à gauche Recommended Yes Yes

… And Core data needs to follow the approved WMO data formats (i.e., BUFR4); Recommended data can provide alternative encodings. … And we don’t want to overwhelm data consumers with more data than they’re expecting (especially for those data consumers that are migrating from GTS) – e.g., high-frequency observations, high-resolution.

${\color{green} F/ Metadata \ management \ and \ catalogue \ synchronization}$

• Different contents in GDCs: https://jira.wmo.int/browse/WI-55 • Persistent presence of outdated topics in catalogues • Lack of synchronization between catalogues • WCMP2 records rejected by GDC WCMP2 records produced using the wis2box dataset editor are rejected by the GDC when the data-policy is recommended · Issue #935 · World-Meteorological-Organization/wis2box. Can we have a predefined license for ICAO ? • Recommendations to improve metadata quality and alignment across nodes Also see https://github.com/wmo-im/wis2-implementation-transition/issues/4

G/ GC retention period.

• Add optional cache-duration attribute to notification message Add optional cache-duration attribute to notification message. · Issue #71 · wmo-im/wis2-notification-message • Add identification of Global Caches to WNM Add identification of Global Caches to WNM · Issue #156 · wmo-im/wis2-notification-message, and add identification of Global Caches to WNM · Issue #190 · wmo-im/wis2-guide

H/ Publishing historical data in WIS2.

Should historical data, such as climate and hydrology time series, be published through WIS2, and if so, how can this be done effectively?

${\color{orange}I/ Handling \ of \ discontinued \ data}$

• Procedures for retiring datasets: o Update WCMP2 notification with an end-date? o Send WCMP2 delete notification? • Best practices and potential impact on subscribers (update the cookbook?)

J/ Fixed IP address for Global Services ${\color{red}{(relevant\ to \ ET-WISOP)}}$

• Some GS are using multiple IP addresses (e.g, JMA is using 6 IP addresses)
• Is there a recommendation or best practice for Global Services to use a single fixed IP address or a narrow range of known IPs?
• Agreement on a recommendation or guideline for IP management by Global Services.
• Possibly draft a cookbook section or implementation note on IP address practices and expectations for both publishers and consumers.

K/ Sensor centres.

• Identify a list of candidate sensor centres • Identify responsible parties and milestones.
See also: https://github.com/wmo-im/wis2-sensor-global-cache

L/ WIS2 Node operations and deployment.

1/ EUMETSAT DCP integration.

M/ WIS2 monitoring and incident management

  • Monitoring enhancements
    • Automated ticket creation
    • Monitoring notification process (criteria TBD)
    • Metrics tracking and analysis
  • Gateways: metrics and monitoring dashboards

N/ Security and governance.

  • Absence of defined timeliness requirements for WIS2 Nodes and Global Services in the Technical Regulations
    -Should this be addressed in future updates?
    -Cybersecurity concerns:
    -Lack of guidance on handling denial-of-service (DoS) attacks
    -Lack of guidance on cybersecurity

P/ Migration where on RTH is generating bulletins for multiple centres.

[Kai] How do we do the transition if in the GTS era one RTH is generating the bulletins for two centers, whereas in WIS2, both centers will operate their own WIS2 Node. The question being how to ensure that no stations are lost, when the two centers do not migrate to WIS2 at the same time. Do we have both Gateways active in that case?

${\color{green}Q/ Readiness \ for \ metadata \ validation }$ (effective 1-Sep-2025 date to be changed)

[Kai?] The other issue would be we still have several centers that provide data without metadata. And we discussed the issues with the GDC. What steps do we need to take to ensure everything works fine when the metadata check is enabled on September 1st? Here I would find it helpful if the monitor messages also contained the checks that failed (i.e. Missing Metadata ID or topic not matching or no metadata available at all).

R/ Proposed topics and updates for the WIS2 Cookbook.

1/ Subscribing to Dataset Metadata Notifications

  • Topic: How consumers can subscribe to notifications about new or updated datasets via metadata notifications (WCMP2 records).
  • Update Needed: Provide clear examples using MQTT and/or API queries to subscribe to metadata channels (e.g. metadata/#), including filtering by data_status, center_id, or observedProperty.

2/ Understanding Dataset Status (Operational vs Experimental)

  • Topic: How consumers can determine if data is classified as operational, pre-operational, or experimental.
  • Update Needed: Clarify the use of data_status in WCMP2 metadata and how consumers can interpret this field for decision-making. Include examples.

3/ Monitoring Data Availability and Issues via Dashboards

  • Topic: How consumers and data publishers can use existing dashboards to identify issues with data provision.
  • Update Needed: Document dashboard features and how users can interpret metrics, alerts, and status pages. Consider including guidance on interpreting automated tickets or alerts.

4/ Handling Multi-Source or Duplicated Datasets from Regional/Global Centres

  • Topic: Regional or global centers (e.g. IFREMER, future METEOALARM) may act as data integrators, aggregating, quality-controlling, or transforming data from multiple upstream sources.

  • Challenge: This can lead to multiple entries of similar datasets, which may confuse users.

  • Update Needed:

    • Add guidance or best practices for avoiding ambiguity
    • Would providing clear, well-structured metadata that explains the differences between datasets be considered sufficient?
    • Or should there be a coordination mechanism or guidance to avoid duplication and ensure clarity for end users?

S/ Issues & Recommendations identified at the Preparation meeting for WIS2-to-GTS Gateway Test (Feb-2025)

(note – this includes some suggested topics for the Cookbook; no attempt has been made to de-duplicate)

1/ GTS-to-WIS2 Gateways to have consistent CCCC lists.

In Feb 2025, DWD and JMA Gateways were not publishing the same GTS bulletins. Each Gateway should publish the GTS bulletins from an identical set of CCCCs. For the same CCCC, TTAAii and WSI may be different for each Gateway. We won’t try to establish parity between Gateways at the TTAAii and WSI level.

2/ Evolve meteomaroc dashboard into a Sensor Centre that can be provided to Centres intending to retire their MSS so that they can determine if the data they need is being shared.

3/ Extend obligation on WIS2 publishers to include datetime in WNMs to make it easier for WIS2-to-GTS Gateways to name bulletins (according to the GTS file naming convention). Currently datetime is optional: https://wmo-im.github.io/wis2-notification-message/standard/wis2-notification-message-STABLE.html#_1_12_properties_temporal_description

4/ Add new section to the WIS2 Cookbook explaining how can consumers subscribe to notifications about new Datasets (via metadata notifications); and how consumers can see that the data is operational or experimental.

5/ Add new section to the WIS2 Cookbook explaining how can consumers use the dashboards to see if there are issues with data provision.

6/ Add new section to the WIS2 Cookbook explaining how WIS2 client applications should work with in-line and remote data sources – to be consistent with how Global Caches behave (see below).

Global Cache behaviour (at least for de-dwd-global-cache) is:

  1. Look for in-line data (properties.content)

  2. If present, retrieve and decode (based on the properties.content.encoding value); validate data size (properties.content.encoding) and, if present, the integrity check (properties.integrity).

  3. If decode and validation passes, proceed to cache the data [end]. ** Else report error ** (for new metrics).

  4. Attempt to download the data from URL.

  5. If download successful, validate file integrity (properties.integrity). Else report error [end]

  6. If validation successful, proceed to cache the data [end]. Else, report error [end]

7/ Update the metrics definitions to clarify the different meaning of how centre-id is used as a label.

For Global Broker: “centre-id” is the centre-id of the originator of the notification message - which could be a WIS2 Node, a Global Broker, or another Global Service

Note: the mapping between MQTT endpoint (the address subscribed to) and the centre-id is part of the Global Broker configuration.

For Global Cache: “centre-id” is the centre-id of the originator/producer of the data - which will be an origin WIS2 Node

Note: a Global Cache cannot pre-determine a mapping between the dataserver URL (domain-name) and centre-id, because the dataserver address is provided in the WIS2 Notification Message.

8/ Require that Global Caches include properties.cache-id to identify where data is hosted by a Global Cache. The problem with using “dataserver” as a metric label

• A WIS2 Node might use several dataservers (e.g., ca-eccc-msc) which adds complexity to the metrics.
• We’re not really interested in differentiating between a WIS2 Node’s dataservers - only the behaviour of the WIS2 Node as a whole.
• Furthermore, if a WIS2 Node changes its dataserver(s), previous metrics referencing a retired dataserver will remain forever (or until reset).
• For the _dataserver_status_flag metric, this means that it will always look like the server is connected (or always failed) because the metrics are only changed when an attempt to use that dataserver was made.
So - instead of using the host-name as the “dataserver”, we propose to use the centre-id for where the data is hosted - either a WIS2 Node or a Global Cache.
Where the data is hosted by the origin WIS2 Node it is simple to use the centre-id provided in the notification message.
However, Global Caches don’t currently identify themselves - except through the download URL.
Consequently, to help identify when data is hosted in a Global Cache, a Global Cache should add properties.cache-id = « centre-id »
… to every notification message they publish - effectively “signing” the message.

9/ Update labels used for Global Services metrics.

For all metrics relating to retrieval of data (see list) the following labels shall be used:

  1. centre-id … the “centre-id” of the origin WIS2 Node where the data was first published
  2. report-by … the “centre-id” of the Global Service reporting the metric
  3. source … the location where the data is retrieved from: if extracted from the WNM this shall be “wnm” (to be confirmed by @Tom Kralidis), if the WNM includes properties.cache-id use that value, else use the centre-id where the data originated
    This change affects: Global Caches, Gateways and maybe Global Monitor dashboards / Sensor Centres.
    The following metrics relate to data “retrieval” - either in-line from the message or download from a URL
    For Global Cache:
    • wmo_wis2_gc_downloaded_total
    • wmo_wis2_gc_downloaded_errors_total
    • wmo_wis2_gc_dataserver_status_flag
    • wmo_wis2_gc_dataserver_last_download_timestamp_seconds << the last time an attempt was made, successful or not
    • wmo_wis2_gc_integrity_failed_total
    For WIS2-to-GTS gateway:
    • wmo_wis2_wg_downloaded_total
    • wmo_wis2_wg_downloaded_errors_total
    • wmo_wis2_wg_dataserver_status_flag
    • wmo_wis2_wg_dataserver_last_download_timestamp_seconds
    • wmo_wis2_wg_integrity_failed_total

10/ Set retention period for Global Monitors to be a minimum of 50GB to allow for gathering of performance statistics for KPIs.

11/ Implement “blacklisting” of TTAAii CCCC headers in the GTS-to-WIS2 Gateways.

Whitelists for WIS2-to-GTS gateway and blacklists for GTS-to-WIS2 gateway.
In the current situation, when a WIS-to-GTS gateway publishes data onto the GTS, these bulletins will be republished back into WIS2 (albeit in the gts-centric topics used by the two _gts-to-wis2 gateways). We want to avoid this duplication, thereby encouraging (requiring!) data consumers to access data natively published into WIS2 using the “proper” topic hierarchies.
Consequently, we agree that the GTS-to-WIS2 gateways should implement a “blacklist” that directly corresponds to the “whitelist” of TTAAii CCCC used by the WIS2-to-GTS gateways. The blacklist would be used to filter out and discard bulletins that appear on the list.
Secretariat to update their “MSS decommissioning” procedures to include sending the list of TTAAii CCCC headers to the GTS-to-WIS2 gateways operators (DWD, JMA).

12/ Update WIS2 Cookbook (or Guide?) to make it clear to data publishers that properties.gts should only be used in WIS2 Notification Messages that relate to data that may be shared on the GTS with a TTAAii CCCC header. Reiterate that the intention of the properties.gts is to enable the WIS2-to-GTS gateway to map the data it receives into a GTS bulletin for distribution. Inclusion of data in a GTS bulletin from “extra” stations will be problematic.

13/ Amend WIS2 documentation to indicated that, in addition to providing TTAAii CCCC, Members wanting to decommission their MSS also indicate which observation stations are associated with each bulletin header (where that's appropriate).Because that will make it easier to determine what data is being shared prior to their MSS being decommissioned.