Datadog Autodiscovery - kimschles/schlesinger-knowledge GitHub Wiki

Datadog Autodiscovery

Big takeaway: At Fairwinds, we use the Datadog autodiscovery pod annotations in things like cert-manager and external-dns to enable the Prometheus Metrics check so we can get some custom metrics we use for RODD monitors.

Objectives

  1. Articulate the need for autodiscovery
  2. Define autodiscovery
  3. Identify reckoner snippets that enable autodiscovery
  4. Describe the parameters that are required to enable Datadog autodiscovery in a pod annotation
  5. Explain why we use Prometheus metrics in our autodiscovery configuration

1. Why do we need DD AD?

  • Containers can move from host to host
  • Kubernetes pods have unique hashes at the end of the name and are terminated often
  • You don't want to manually setup custom metrics monitoring for containers in a Kubernetes cluster

2. Define AD

  • Datadog autodiscovery ensures the agent looks at all services running in a container, and will apply any metric collection rules it's got saved.
  • From the DD docs:

    The goal of Autodiscovery is to apply a Datadog integration configuration when running an Agent check against a given container.

3. Identify reckoner snippets that enable autodiscovery

From nginx-ingress

        podAnnotations:
          ad.datadoghq.com/nginx-ingress-controller.check_names: |
            ["prometheus", "nginx_ingress_controller"]
          ad.datadoghq.com/nginx-ingress-controller.init_configs: |
            [{}, {}]
          ad.datadoghq.com/nginx-ingress-controller.instances: |
            [
              {
                "prometheus_url": "http://%%host%%:10254/metrics",
                "namespace": "ingress",
                "metrics": ["nginx_ingress_controller_config_last_reload_successful"]
              },
              {
                "prometheus_url": "http://%%host%%:10254/metrics"
              }
            ]

4. Describe the parameters that are required to enable Datadog autodiscovery in a pod annotation

Kubernetes Integrations Autodiscovery Docs

  annotations:
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check_names: '[<INTEGRATION_NAME>]'
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.init_configs: '[<INIT_CONFIG>]'
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.instances: '[<INSTANCE_CONFIG>]'

5. Why are we using Prometheus metrics?

To set up integrations that are not compatible with standard Autodiscovery, you can use an official Prometheus exporter in the pod, and then use the OpenMetrics check with Autodiscovery in the Agent to find the pod and query the endpoint. For example, the standard pattern in Kubernetes is: side car adapter with a node-level or cluster-level collector. This setup allows the exporter to access the data, which exposes it using an HTTP endpoint, and the OpenMetrics check with Datadog Autodiscovery can then access the data. -- Kubernetes Integrations Autodiscovery Docs

OpenMetrics or Prometheus Check Snippet

        ad\.datadoghq\.com/external-dns\.instances: |
          [
            {
              "prometheus_url": "http://%%host%%:7979/metrics"\,
              "namespace": "external-dns"\,
              "metrics": ["*_errors_total"]\,
              "max_returned_metrics": "1000"
            }
          ]