Symptoms

You are missing a set of metric values for one metric from one target instance, but still may receive metric values from others.

You are seeing the warning message

Attempt to merge metric xxx, but types are deviating: ...

(before Promregator version 0.6.0).

You are seeing the warning message

Scraping resulted in a collision of metric types for metric with name xxx. 
The conflicting types provided were X and Y. 
The data with type X is kept, while the other metrics values will be dropped. 
For further details see also https://github.com/promregator/promregator/wiki/Multiple-Type-Declarations-of-a-Metric-Cause-a-Collision-in-Single-Endpoint-Scraping-Mode ; Details: ...

(Promregator version 0.6.0 and later).

Background / Root Cause

Prometheus requires metrics to declare a type for it. In a nutshell, as of writing, Prometheus supports the following four types:

• Counter
• Gauge
• Histogram
• Summary

Each metric may have multiple values, which, however, must be separated by deviating tags.

By providing tags to metrics providing knowledge about the data's origin, Promregator tries to ensure that metric values are not overwriting each other. However, Prometheus requires that metrics keep their type.

If Promregator is running in Single Endpoint Scraping mode and scraped more than one target instance, then the following race situation may occur:

• Target Instance 1 emits a metric and declares in its metrics data that it is of type X.
• Target Instance 2 emits the same metric (with the same name), but declares that it has type Y (with Y unequal to X).

If Promregator is running in Single Endpoint Scraping mode then, Promregator is requested by Prometheus to only expose a metric with one single type only (there cannot be two types at the same point in time). Given the specifications and definitions, Promregator is unable to do so.

Solution

In this situation, Promregator reacts like this:

1. Promregator takes over the first (arbitrary) definition; for the sake of simplicity let us assume that this is the declaration provided by Target Instance 1.
2. On detecting the second (contradicting) declaration, Promregator emits the warning message as mentioned above.
3. Promregator drops the values of the target instance providing the contradicting metric type.

Promregator repeats this behavior for all target instances providing a contradicting metric type.

Note that that the dropping and issuing of the warning is only performed for this single metric! Other (unaffected) metric values provided by a target instance are still copied and provided as usual.

This approach is only a protection mechanism, preventing Promregator to send metric data, which otherwise would be rejected by Prometheus anyway.

There are two known solutions to this problem:

1. You may switch from Single Endpoint Scraping mode to Single Target Scraping mode. A major aspect to the situation is that Promregator must provide a combined (merged) view of the metrics' data to Prometheus, but must write that into a single document. If Promregator is running in Single Target Scraping mode, Prometheus requests multiple documents (one for each target instance) and thus is not forced to merge the metrics data.

   Note that this approach only will shift the type collision from Promregator to Prometheus. The type collision must be handled there appropriately as well.

2. Ensure that the type collision is resolved. Generally, there are two cases to consider:

   1. The type simply is provided erroneously: That is typically the case if an old version of a client library in one of the target instances gets used, or additional metrics data is exposed, which is done unintentionally.

   2. The metrics have different semantics, but (erroneously) take the same name: Make sure that the two target instances in conflict expose metrics with different names. That is the only proper way of resolving this kind of issue, as otherwise, when analyzing the metrics data in Prometheus later on, you will suffer from big headache either.

References

• This page is based on a discussion at https://github.com/promregator/promregator/issues/104.