Invalid Version In Content Type of OpenMetrics Endpoints - promregator/promregator GitHub Wiki
Background
With the advent of the OpenMetrics format, a new Content Type for scraping metrics has been introduced to the Prometheus world. In old times, there was only the Content Type
text/plain; version=0.0.4; charset=utf-8
whereas the new OpenMetrics format has the Content Type
application/openmetrics-text; version=1.0.0; charset=utf-8
Note the version number here reads 1.0.0
. The OpenMetrics Specification does not foresee any other version identifier so far - in particular there is no such version which has a major version number of zero (0
).
The Problem
Apparently, due to an initial bug at Prometheus, there are some clients out there, which think that they should return a content type which contains the version identifier of 0.0.1
. That is wrong.
Reaction of Promregator
Promregator tries to remain fault-tolerant in this direction: It will write a warning into its logs reading along the lines of
The implementation at https://example.bogus and instance 5 returned an invalid content type
by specifying an invalid version identifier 0.0.42 of the OpenMetric format. Promregator
is guessing that you mean version 1.0.0. Please fix your Prometheus client library!
It will assume that the client has meant to send version identifier 1.0.0. This may cause issues in the future, if a new version of the specification is provided.
What You Should Do
- Check the implementation of the Prometheus' client library the scraping target uses. Perhaps there is already a newer version available that addresses this issue. If that is the case, upgrade your client library to ensure that you are compliant to the specification.
- In case that there is no fixed version available, check the bug tracker / issue management tool of your Prometheus' client library. Perhaps the issue is already known but not fixed yet. Report there that you are also affected from this issue.
- In case the issue is not known to the Prometheus' client library yet, create a new bug report for them. Refer to this page to help you explaining what is wrong. Ask them to fix it on next occasion.
How to Prevent Log Flooding
Until the issue is fixed at your Prometheus' client library and you were able bump to that version, you may disable the warning in your Promregator's log files by setting the log level of logger named org.cloudfoundry.promregator.fetcher.CFMetricsFetcher.wrongVersion
to ERROR.
Fixes in Client Libraries
- Java Client: https://github.com/prometheus/client_java/issues/702
- Spring-Boot (Java): https://github.com/spring-projects/spring-boot/issues/28130
- Python Client: https://github.com/prometheus/client_python/issues/901