Service Accounts authentication is controlled via keys. In a vanilla GCP project, default time to expire for a key is a bit less than 8000 years. Practically that means, if your key is leaked and you missed that event, it can be abused for a long time.

An ideal way to configure Service Accounts keys in Hybrid is to use Workload Identities, that doesn't expose the keys as well as rotates them automatically and transparently, thus reducing the blast radius in case of compromise. https://cloud.google.com/apigee/docs/hybrid/v1.5/enable-workload-identity

For non-GKE clusters which do not support Workload Identities, the next best is to rotate keys frequently.

Correct way to manage this security risk is to set up an org-level of your company for a short time period. A week or so. You can do it daily.

Apigee Hybrid allows granular control of Service Accounts. You can use different Service Accounts for different clusters, production/non-production deployments. Every apigee environments can have different Service Account.

Mart and Connect use same secret. A secret for apigee-runtime [distributed trace] is optional.

ahr-sa-ctl sa-keys-* collection of command automates this task as well as easies troubleshooting and management of SA Keys.

We configure service account key file path via runtime configuration yaml file (a.k.a, overrides file).

Service accounts are created at the GCP IAM/Service Accounts page. Then the required component roles are bound to them.

Keys are created either manually, or using resident create-service-account utility, or by ahr-sa-ctl command. https://cloud.google.com/apigee/docs/hybrid/v1.5/sa-about

At the Kubernetes side, apigeectl configures CRDs and Kubernetes secrets to associate IAM Service Accounts with imported keys

?. Prepare a bash file you can source with variables required to run the commands.

cat <<EOF > source.env
export AHR_HOME=~/projects/ahr
export PATH=$AHR_HOME/bin:$PATH

export PROJECT=<project-id>

CLUSTER=<cluster-name>
CLUSTER_ZONE=<gcp-zone>

export ORG=$PROJECT
export SA_DIR=<service-accounts-key-files-directory>

# fetch/configure cluster authentication context
# kubectl config use-context <context-name>
# for GKE cluster, it would be different for other clouds/on-prem
gcloud container clusters get-credentials $CLUSTER --zone $CLUSTER_ZONE --project $PROJECT
EOF

?. Source it

source source.env

ahr-sa-ctl sa-keys-* commands allow you easily implement key rotation as well as troubleshoot and manage SA keys.

These commands are run against a live system. sa-keys-report commands need access to gcloud and kubectl. The sa-keys-rotate expects SA_DIR environment variable to be configuredf,as it needs to use $SA_DIR folder as an interim storage, but no other configuration files are requied. These commands are meant to work 'autonomously' with a minimum of config knowledge for easy of use as cron job.

Rotation process as executed by sa-keys-rotate commands are three steps:

• Download keys
• Update secrets
• Rolling update on all pods

Removal of expired or unneeded kays can be done using sa-keys-delete.

The sa-keys report commands shows key-related and key configuraton information.

It iterates across Hybrid org-level components, as well as each configured environment, collects information, then reports it.

Column DTE is a date to expire. In the example below, the key expired 37 days ago. Kind is a Hybrid CRD component, either ApigeeDeployment, which is implemented as RecordSet or ApigeeTelemetry, which is implemented as DaemonSet.

We need this information, because rolling restart commands for those resources are different.

Usually default names for Service Account Ids are used, yet in general case, those can be custom and multiple per project. Thus, specific values are reported.

NOTE: The cryptic-looking part of the component structured RESOURCE or SECRET_NAME properties are SHA values for either $ORG or $ORG$ENV. The purpose is to fit the total catenated name in an 63 char Kubernetes resource name maximum length limit. https://kubernetes.io/docs/concepts/overview/working-with-objects/names/

The sa-keys-rotate command is a command that automatically rotates keys in every Hybrid component. It is meant to be configured either as a Kubernetes cronjob or your ifrastructure job.

You can execute the command using --debug and --dry-run options to see how internally your components, service-accounts and keys are related without applying changes.

The rotate command uses rolling update commands to restart specific components with zero-downtime.

A service account can have up to 10 keys. Additionally, your Cloud Organization administrator might configer total maximum number of keys per project limit. This requires explicit management of the keys.

The sa-keys-list command list existing list for either all or a specific SA.

You can use sa-keys-delete --filter option to remove keys that are not needed anymore.

You can use --dry-run option to verify which keys will be affected.

$ ahr-sa-ctl sa-keys-delete all
--filter=<field><comp><iso-date> option must be provided. Example:
  ahr-sa-ctl sa-keys-delete all --dry-run --filter "EXPIRES_AT<$(date -Ins)"
  ahr-sa-ctl sa-keys-delete all --dry-run --filter "EXPIRES_AT<2021-07-07T23:59

Command reports date command is not GNU:

ABEND. date command is not GNU.
       For OSX  set up coreutils and use"
       <your local bin with mv-ed date>/PATH to substitute date with gdate"
       instead of the resident one."
       For Windows, use cygwin."

OSX is not a Linux. date in Linux the date is GNU and date in osx is a bastardised bsd.

A good clean easy example way to make the utilites work at OSX is to homebrew in your local folder + $PATH

Set up coreutils and use /PATH to substitute date instead of the resident one.

Same applies for a version of Bash. Apple uses ancient version of bash and will upgrade it due to licensing issues. Use your favorite packager [homebrew?] to install bash 4.x or 5.x. Then add them to your path. As utilities use portable version of shebang '#!/usr/bin/env bash`, they will pick up a correct version of bash.

If you run sa-keys-* command on an unsupported version of Hybrid, your output would have many dashes (-) as the names of secrets were different for those version. It does not make sense to rotate keys for those versions. Upgrade Hybrid to any supported version, then run -rotate or -report commands.

Example for 1.1 output.

$ ahr-sa-ctl sa-keys-report
Processing: emea-cs-hybrid-demo2:AD/apigee-connect-agentError from server (NotFound): secrets "apigee-connect-agent-emea-cs-hybrid--6fde6b2-svc-account" not found
 emea-cs-hybrid-demo2:AD/apigee-martError from server (NotFound): secrets "apigee-mart-emea-cs-hybrid--6fde6b2-svc-account" not found
 emea-cs-hybrid-demo2:AD/apigee-watcherError from server (NotFound): secrets "apigee-watcher-emea-cs-hybrid--6fde6b2-svc-account" not found
 prod:AD/apigee-synchronizer prod:AD/apigee-udca prod:AD/apigee-runtime test:AD/apigee-synchronizer test:AD/apigee-udca test:AD/apigee-runtime AD/apigee-metrics-apigee-telemetry-appError from server (NotFound): secrets "apigee-metrics-svc" not found
 AD/apigee-metrics-apigee-telemetry-proxyError from server (NotFound): secrets "apigee-metrics-svc" not found
 AT/apigee-telemetryError from server (NotFound): secrets "apigee-logger-svc" not found

KIND  RESOURCE                                          SECRET_NAME  SA_ID  SA_KEY_ID  DTE  EXPIRES_AT
AD    apigee-connect-agent-emea-cs-hybrid--6fde6b2      -            -      -          -    -
AD    apigee-mart-emea-cs-hybrid--6fde6b2               -            -      -          -    -
AD    apigee-watcher-emea-cs-hybrid--6fde6b2            -            -      -          -    -
AD    apigee-synchronizer-emea-cs-hybrid--prod-b9d1ce8  -            -      -          -    -
AD    apigee-udca-emea-cs-hybrid--prod-b9d1ce8          -            -      -          -    -
AD    apigee-runtime-emea-cs-hybrid--prod-b9d1ce8       -            -      -          -    -
AD    apigee-synchronizer-emea-cs-hybrid--test-6011acb  -            -      -          -    -
AD    apigee-udca-emea-cs-hybrid--test-6011acb          -            -      -          -    -
AD    apigee-runtime-emea-cs-hybrid--test-6011acb       -            -      -          -    -
AD    apigee-metrics-apigee-telemetry-app               -            -      -          -    -
AD    apigee-metrics-apigee-telemetry-proxy             -            -      -          -    -
AT    apigee-telemetry                                  -            -      -          -    -
bash-5.0$