configure prometheus service account to use more than more iam role to scrape targets in multiple aws accounts - jvcastaneda/DevopsToolingConfig GitHub Wiki

To scrape EC2 targets across multiple AWS accounts with a single Prometheus server running in EKS, you essentially give Prometheus one “primary” IRSA role that can assume additional roles in each target account, and then tell Prometheus to use those per-account roles in its EC2 service-discovery blocks.

1. In each target account, create a cross-account role

• Trust policy: allow your EKS cluster’s IAM role (the “primary” Prometheus role) to assume it.
• Permissions: at minimum ec2:DescribeInstances (and ec2:DescribeAvailabilityZones if you want AZ labels).

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::<EKS-account-ID>:role/PrometheusPrimaryRole"
    },
    "Action": "sts:AssumeRole"
  }]
}

citeturn0search0

2. In your EKS account, create the “primary” Prometheus IAM role

• Annotate the Prometheus service account with it via IRSA:

  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: prometheus
    namespace: monitoring
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<EKS-account-ID>:role/PrometheusPrimaryRole

• Attach a policy allowing it to assume all the cross-account roles you just created:

  {
    "Version":"2012-10-17",
    "Statement":[{
      "Effect":"Allow",
      "Action":["sts:AssumeRole"],
      "Resource":[
        "arn:aws:iam::<account-A-ID>:role/EC2DiscoveryRole",
        "arn:aws:iam::<account-B-ID>:role/EC2DiscoveryRole",
        …
      ]
    }]
  }

citeturn1search0

3. Tell Prometheus about each account in prometheus.yml
Use multiple ec2_sd_configs, one per account, pointing at the cross-account role ARN:

scrape_configs:
  - job_name: aws-account-A
    ec2_sd_configs:
      - region: us-east-1
        role_arn: arn:aws:iam::<account-A-ID>:role/EC2DiscoveryRole
        refresh_interval: 60s

  - job_name: aws-account-B
    ec2_sd_configs:
      - region: us-west-2
        role_arn: arn:aws:iam::<account-B-ID>:role/EC2DiscoveryRole
        refresh_interval: 60s

When Prometheus runs, the AWS SDK will:

1. Use its IRSA-mounted WebIdentity token to assume PrometheusPrimaryRole.
2. For each ec2_sd_config.role_arn, call STS to assume the target account’s role.
3. Call EC2 DescribeInstances in that account and pick up the targets.
   citeturn1search4turn0search0

Add your annotation under serviceAccount and drop your extra scrape configs in a Secret:

# values.yaml
prometheus:
  prometheusSpec:
    serviceAccountName: prometheus
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::<EKS-account>:role/PrometheusPrimaryRole

    additionalScrapeConfigs:
      name: extra-scrapes
      key: prometheus-additional.yaml

# extra-scrapes Secret
apiVersion: v1
kind: Secret
metadata:
  name: extra-scrapes
  namespace: monitoring
stringData:
  prometheus-additional.yaml: |
    - job_name: aws-account-A
      ec2_sd_configs:
      - region: us-east-1
        role_arn: arn:aws:iam::<accountA>:role/EC2DiscoveryRole

    - job_name: aws-account-B
      ec2_sd_configs:
      - region: us-west-2
        role_arn: arn:aws:iam::<accountB>:role/EC2DiscoveryRole

That way, a single Prometheus pod (and IRSA role) can reach into any number of AWS accounts—each EC2 SD block will assume its own role.