3. Handling secrets in GCP - grzzboot/pingpong-service GitHub Wiki

I think it's fair to say that pretty much all systems have some need for secrets to be configured in order for them to operate successfully. It can be a database password, a service admin account or something else that is needed by the system in runtime yet must kept safe and locked in under all other circumstances. There are numerous approaches of how to handle secrets and of course a whole fleet of solutions that you can purchase to do the job. In this article we're going to go through, and see examples of two of the ships that Google Cloud adds to the fleet and how they can be integrated into your processes, be they in the Cloud or somewhere else.

To keep you awake and try to make this fun we'll be using GKE to see our secrets being used in runtime. You can of course apply the secrets in other contexts as well, even on an On-Premises solution that has nothing to do with GKE, GCP or Google at all; - these are SaaS solutions.

The two different solutions we'll explore are:

Cloud Key Management (KMS), denoted as Cryptographic Keys under the Security tab in the Web Console main menu
Secret Manager, also under the Security tab in the Web Console main menu

Kubernetes Secret objects

Before we do anything more we should set up a small cluster, since we'll be exploring these features using GKE. You can refer to the article about Kubernetes-Engine in this wiki to get help on setup, or if you feel comfortable to do it by hand that's fine too. You'll need a single node with at least one 1 CPU.

Inside Kubernetes the containers gets access to secret data by referring Kubernetes Secret objects that contains the actual data. This means that in order for a secret value to become available to a service in GKE we need to first contain it in a Secrets object that we create. Just to show how this works I've set up a super-simple first example that uses a hardwired secret, committed into the source code effectively making it a non-secret.

A hardwired secret, as part of code, just to show the basics.

Go to the hardwired-secret-folder under the k8s-folder of the pingpong-service-secrets module and perform a deploy as per usual. This will launch a deployment without any exposure so we'll need to port-forward to try it out. But before we do that let's take a look inside to see what's going on and how we can determine if things are working as we anticipate.

If you take a look inside the PingResourceController.java class of the pingpong-service-secrets module you'll see the that the endpoint is really simple; - it takes no parameters and simply performs a call to the service class.

(Omitted some logging lines for readability)

@GetMapping(path = "/ping")
public PingResource ping() {
  ...
  PingResource pingResource = new PingResource(pingPongService.ping().getMessage());
  ...
  return pingResource;
}

The service class in its turn contains the following:

public PingEntity ping() {
  StringBuilder sb = new StringBuilder(MESSAGE_BASE);
  sb.append(", the secret is " + secretsConfig.getSecret());
  return new PingEntity(sb.toString());
}

As you can see it produces a response string by reading a secret from a SecretsConfig class. The SecretsConfig class is really simple and looks like this:

(Omitted annotations for readability)
...
@ConfigurationProperties(prefix = "pingpong.secrets")
public class SecretsConfig {

  private String secret;

}

This @ConfigurationProperties(prefix = "pingpong.secrets") is a Spring Boot way of saying that; - read all properties defined in the application.properties file(s) that starts with pingpong.secrets and has a matching property (nested or not) in this class. So if Spring Boot finds a property in the application.properties file(s) that is defined as pingpong.secrets.secret it will populate the secret property of this class. If it finds a property called pingpong.secrets.password it will do nothing because there is no property with that name in the SecretsConfig class.

So by inspecting the application.properties file we can finally understand what will happen when we start the application. The relevant part in the file is this one:

pingpong.secrets.secret=${PINGPONG_SECRETS_SECRET:default and unconfigured}

The somewhat cryptical notation is used to specify the usage of an environment variable when present, or a default value when not; - ${ENVIRONMENT_VARIABLE:defaultValue}. So when the Spring Boot service is fed with an environment variable like -DPINGPONG_SECRETS_SECRET=whatever it will use whatever as value. If no such environment variable is available it will use the default value default and unconfigured.

Startup the service locally in your IDE or using you console and then navigate to http://localhost:8080/ping. Since you haven't configured any environment variable (I suppose?) you should see the following output:

{
  message: "Pong, the secret is default and unconfigured"
}

Ok, stop the local service, if you haven't already, and port-forward to the pod that was created when you deployed above. Again, invoke the same URL and you should see:

{
  message: "Pong, the secret is hardwired in env-file. Not recommended to commit a secret like this!"
}

Wooaah!? How did that happen? Well, if you look inside the kustomization.yaml of the hardwired-secret folder you'll see these lines:

secretGenerator:
- envs:
  - pingpong-secrets.env
  name: pingpong-secrets

This is a Kustomize specific notation that generates a Kubernetes Secret object based on a file. The pingpongs-secrets.env is the file and the name of the generated Kubernetes Secret object is pingpong-secrets.

The pingpong-secrets.env file can be found right next to the kustomization.yaml file and it looks like this:

PINGPONG_SECRETS_SECRET=hardwired in env-file. Not recommended to commit a secret like this!

If you repeat the kustomize build . command but without applying the result you'll get the YAML-blob and you can see what a Kubernetes Secret object looks like. It's something like this:

apiVersion: v1
data:
  PINGPONG_SECRETS_SECRET: aGFyZHdpcmVkIGluIGVudi1maWxlLiBOb3QgcmVjb21tZW5kZWQgdG8gY29tbWl0IGEgc2VjcmV0IGxpa2UgdGhpcyE=
kind: Secret
metadata:
  labels:
    component: pingpong
  name: pingpong-secrets-fkd6c6ccg2
  namespace: pingpong
type: Opaque

That long line there contains the key and a base64 encoded variant of the data part from the pingpong-secrets.env file. This is fed into, or connected to if you will, the Kubernetes Container object by reference in the deployment.yaml file:

envFrom:
  - secretRef:
      name: pingpong-secrets

The envFrom indicates how the key/value-pair will be fed to the container; - as an environment variable. And as we stated above Spring Boot will look for an environment variable with the proper name, PINGPONG_SECRETS_SECRET, and use it when present. What we're sort of doing is setting the -D-flag, although in a very complicated way it may seem...

Complicated or not, this opens up for us to store secrets in any way as long as we can obtain them during deploy and create a Secret from them. We can completely hide real and important secrets from essentially everyone except for the deployment-pipeline and perhaps those responsible of defining the secret in the first place.

The remainder of this article will cover a set of ways to store and make use of stored secrets in this way.

Cloud Key Management (KMS) encrypted secrets

KMS is based on the use of cryptographic keys with which you encrypt and decrypt your secrets. Having the secrets securely encrypted means that you can place them in your source code. Only someone with decrypt IAM privileges to your KMS resource can decrypt it.

To create a keyring and key do the following:

gcloud kms keyrings create pingpong-secrets-keyring \
  --location=global

gcloud kms keys create pingpong-secrets-key \
  --keyring=pingpong-secrets-keyring \
  --location=global --purpose=encryption

Under the pingpong-service-secrets module and k8s folder there is a folder called kms-secret. There you'll find a prepared text-file called pingpong-secrets.txt. Now you can encrypt the contents of that file by performing the following KMS encrypt command:

gcloud kms encrypt \
  --key=pingpong-secrets-key \
  --keyring=pingpong-secrets-keyring \
  --location=global \
  --plaintext-file=pingpong-secrets.txt \
  --ciphertext-file=pingpong-secrets.env.encrypted

The outcome of this should be a file called pingpong-secrets.env.encrypted which, if you open it, only contains unreadable crap. This file can be committed in your source repository as long as you never, of course, commit the key to decrypt it. You're not gonna be able to do that cause this one is stored in KMS, but still...

To decrypt the file and use it in the deployment do the following:

gcloud kms decrypt \
  --key=pingpong-secrets-key \
  --keyring=pingpong-secrets-keyring \
  --location=global \
  --plaintext-file=pingpong-secrets.env.decrypted \
  --ciphertext-file=pingpong-secrets.env.encrypted

If you inspect the kustomization.yaml file you'll see that it points out a file with the name of the decrypted file:

secretGenerator:
- envs:
  - pingpong-secrets.env.decrypted
  name: pingpong-secrets

Go ahead and deploy from the kms-secret folder. Do a port forward to the new instance and check the http://localhost:8080/ping URL again. You see the message below.

{
  message: "Pong, the secret is encrypted using KMS. This way it CAN be committed safely."
}

The process above where we create keyring, key and then encrypt data is of course not normally done at the same time as the decryption. We just did it here to learn the steps. Normally the keyring and keys are created once and to be used for a large set of secrets. Then in a build pipeline the decrypt is performed at a later time. So it's usually both different actors that do the operations and different contexts in which they are performed.

Speaking about actors...

IAM and KMS

Since you, for this demo project, is most likely the project Owner you get to do anything! This is normally not the case and certainly not the case you want it for a production project with production secrets. Then access to these resources should be restricted. This can be done using IAM operations on the keyring or just the key. Since we can't be involving all your friends in this we need some sort of other account to try this on and therefore we'll create and use a service account. Use the following command to create a service account and a JSON key for secret IAM evaluation (we'll be using it further down below as well).

gcloud iam service-accounts create pingpong-secrets-iam-sa \
    --description="A service account for evaluating IAM and secrets" \
    --display-name="pingpong-secrets-iam-sa"

gcloud iam service-accounts keys create ./pingpong-secrets-iam-sa-key.json \
  --iam-account pingpong-secrets-iam-sa@pingpong-site1-gcp-demo.iam.gserviceaccount.com

Ok, this created a key that we can use to "become" this service account. So we'll be toggling a bit between your normal identity and this service account identity just to see how it works. This will require you to perform the gcloud auth login and identify yourself from time to time. When you are asked to login as the service account you'll perform gcloud auth activate-service-account --key-file=pingpong-secrets-iam-sa-key.json. Let's become the service account first.

Then, try the decrypt again:

gcloud kms decrypt \
  --key=pingpong-secrets-key \
  --keyring=pingpong-secrets-keyring \
  --location=global \
  --plaintext-file=pingpong-secrets.env.decrypted \
  --ciphertext-file=pingpong-secrets.env.encrypted

If you've successfully managed to "become" the service account you should get this error:

ERROR: (gcloud.kms.decrypt) PERMISSION_DENIED: Permission 'cloudkms.cryptoKeyVersions.useToDecrypt' denied on resource 'projects/pingpong-site1-gcp-demo/locations/global/keyRings/pingpong-secrets-keyring/cryptoKeys/pingpong-secrets-key' (or it may not exist).

And that's because the service account doesn't have access to the key. We can change that by becoming you again and then running this command:

gcloud kms keys add-iam-policy-binding pingpong-secrets-key \
  --keyring=pingpong-secrets-keyring \
  --location=global \
  --member=serviceAccount:pingpong-secrets-iam-sa@pingpong-site1-gcp-demo.iam.gserviceaccount.com \
  --role=roles/cloudkms.cryptoKeyDecrypter

This will make the specific service account able to decrypt (nothing else) using this key. It won't be able to encrypt. Become the service account again and verify that decrypt now works and that encrypt doesn't (commands are above).

It's time to become yourself again!

Secret Manager stored secrets

Using the Secret Manager you don't need to store anything in source code, you'll fetch the secret as such on demand, typically from a Build pipeline.

Go to the secret-manager-secret folder under k8s. There you'll find a pingpong-secrets.txt file again and to make a secret out of this one just run the following command:

gcloud secrets create pingpong-secrets-secret \
  --data-file=./pingpong-secrets.txt

This will create a new secret and a new secret version. The secret itself (the file) is always access by pointing out a version, not just the secret. You can point out latest if you always want latest but some recommend you not to. To access the secret run the following:

gcloud secrets versions access 1 \
  --secret=pingpong-secrets-secret > pingpong-secrets.env.managed

Or...

gcloud secrets versions access latest \
  --secret=pingpong-secrets-secret > pingpong-secrets.env.managed

The last bit there will pipe the result into a file called pingpong-secrets.env.managed and of course this is used when deploying. Go ahead and deploy the service, port-forward and check the result from the /ping endpoint. You should see:

{
  message: "Pong, the secret is stored using Secret Manager. No need to commit anything."
}

Secret Manager and IAM

Similarly, as with KMS keyrings and keys, a secret can also be modified w.r.t. access using IAM settings. We'll reuse the service account from above (see KMS if you don't have it). Copy the key that was generated into this folder cp ../kms-secret/*.json . Then become the service account and try to access the secret without piping it to a file this time:

gcloud secrets versions access 1 \
  --secret=pingpong-secrets-secret

You'll get an error very similar to the one for KMS.

ERROR: (gcloud.secrets.versions.access) PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/pingpong-site1-gcp-demo/secrets/pingpong-secrets-secret/versions/1' (or it may not exist).

Adding the service account to the IAM config for this secret is also similar to the KMS way, use this command after becoming yourself:

gcloud secrets add-iam-policy-binding pingpong-secrets-secret \
    --member=serviceAccount:pingpong-secrets-iam-sa@pingpong-site1-gcp-demo.iam.gserviceaccount.com \
    --role=roles/secretmanager.secretAccessor

Then become the service account again and verify that the access works.

Using Cloud Key Management (KMS) or Secret Manager in a build pipeline

KMS and/or Secret Manager can be used internally if you use GCP for hosting purposes or as a SaaS-solution if you run your services elsewhere, even On-Prem.

External build pipelines

External build pipelines, like a declarative Jenkinsfile for instance, can make use of KMS or Secret Manager by using the approach of the service accounts like we displayed above. The two roles used there are the recommended ones to use.

It's generally quite time consuming to set IAM policies for every single secret or key/keyring that you have (it's a bit easier with the keys/keyrings since you often use them for multiple secrets) so often you'll want to give the service account access on project level. After all it's a rather likely scenario that the service account will need access to all secrets/keys/keyrings anyway if it is for your build server.

When creating service accounts then create them for services, don't create them for individual functions. So if you have a service account for a Jenkins server and it needs access to KMS, Secret Manager and let's say GKE, then give all these accesses to ONE Jenkins specific service account. Don't spread the accesses over several different service accounts that you have to configure for Jenkins. This makes it easier to simply delete the service account if you decide to get rid of your Jenkins server.

GCP Native build pipelines (Cloud build)

To get a taste of what it is like to use KMS and/or Secret Manager in a build pipeline we'll use two cloudbuild.yaml files and run them in Cloud Build.

Cloud Build and KMS

When the Cloud Build API is configured it creates a IAM member for the project automatically. This member can be assigned the same roles as the service account created above to be able to use KMS and Secret Manager. The easiest way to do this is using the Web Console and you'll find the member under IAM & Admin > IAM. It has an email similar to 1234567890@cloudbuild.gserviceaccount.com. Edit that member and add the role labeled Cloud KMS CryptoKey Decrypter. Also, since we're going to deploy onto a GKE cluster we need to have access to list and affect the Kubernetes stuff. The easiest way to get that access is to assign the role Kubernetes Engine Developer. It's an over-grant but in this showcase it's the easiest way.

Below instruction of how to perform the Cloud Build is a workaround! I wanted to use the gcr.io/cloud-builders/kubectl builder to generate YAML with kustomize and then apply it, but the kubectl built in kustomize uses an old notation of in particular the secretGenerator which makes the kustomization.yaml from the kms-secret folder incompatible. Therefore I created a subfolder just for the Cloud Build part where the notation in the kustomization.yaml is adjusted to match the kubectl version of kustomize. This might change over time! If you want to build a pipeline using kustomize I recommend that you create your own builder. See Cloud Builders Community - Kustomize

Go inside the cloudbuild subfolder of kms-secret folder and run cp ../pingpong-secrets.env.encrypted . or copy the file by hand. We need it in place.

Undeploy anything already in your cluster to get a clean sheet.

Then, inside the cloudbuild folder you have a cloudbuild-kms.yaml file that you can run by typing gcloud builds submit --config=cloudbuild-kms.yaml. This will submit the folder (including necessary YAML-files and the secret) and execute some steps to decrypt the secret and then deploy the service.

Use watch kubectl get pods -n pingpong to monitor the namespace of the GKE cluster and you should see the service start up once the Deploy step is finished.

NAME                                   READY   STATUS    RESTARTS   AGE
pingpong-deployment-78696f8c7b-5lmjm   1/1     Running   0          55s

Then you can make a port-forward and inspect the http://localhost:8080/ping endpoint and it should say, as before:

{
  message: "Pong, the secret is encrypted using KMS. This way it CAN be committed safely."
}

Cloud Build and Secret Manager

This part is fairly similar to the KMS one. We need to give the secretAccessor privilege to the IAM account for Cloud Build (see above). This is easiest done in the Web Console by adding the role labeled Secret Manager Secret Accessor.

Again to make the Cloud Build deploy step work I made the same workaround as above...

Undeploy anything already in your cluster to get a clean sheet and then go inside the cloudbuild subfolder of secret-manager-secret.

Then, inside the cloudbuild folder you have a cloudbuild-secret-manager.yaml file that you can run by typing gcloud builds submit --config=cloudbuild-secret-manager.yaml. This will submit the folder (including necessary YAML-files and the secret) and execute some steps to decrypt the secret and then deploy the service.

Use watch kubectl get pods -n pingpong to monitor the namespace of the GKE cluster and you should see the service start up once the Deploy step is finished.

NAME                                  READY   STATUS    RESTARTS   AGE
pingpong-deployment-b845f577b-6dx8c   1/1     Running   0          14s

Then you can make a port-forward and inspect the http://localhost:8080/ping endpoint and it should say, as before:

{
  message: "Pong, the secret is stored using Secret Manager. No need to commit anything."
}