Auto-unseal Vault in Kubernetes environments

In this guide, you will learn how to set up Vault to unseal itself in a Kubernetes environment, avoiding dependency of other services as KMS cloud providers, ensuring a secure startup process.

NOTE: This guide assumes that Vault is already installed and initialized with its access keys. If you haven't set it up yet, you can follow this link to do so.

In this guide, Vault will have the following access keys with a key-threshold of 3.

{
  "unseal_keys_b64": [
    "JDM9anYtanlo5iNYJfmCCIa5jsGJjVaKxYBdn98YQYmJ",
    "IKAfWjbA2l41mb4JwPJdSl+7NcM3ZwHpMr2JvyCpZTdl",
    "Nk1W+oDisOL7G6ad88ULnV0ZHRahTqe5as76LjbI43s0",
    "5mGWfOFv7Tj2sKnXRbBhdKDrhGkxmRKbPyUiB8gaBIQR",
    "I4KDsJoVsjGcNtDnKlvAB2ioM7pK8uuW+40PQgAJQP6n"
  ],
  "unseal_keys_hex": [
    "24333d6a762d6a7968e6235825f9820886b98ec1898d568ac5805d9fdf18418989",
    "20a01f5a36c0da5e3599be09c0f25d4a5fbb35c3376701e932bd89bf20a9653765",
    "364d56fa80e2b0e2fb1ba69df3c50b9d5d191d16a14ea7b96acefa2e36c8e37b34",
    "e661967ce16fed38f6b0a9d745b06174a0eb84693199129b3f252207c81a048411",
    "238283b09a15b2319c36d0e72a5bc00768a833ba4af2eb96fb8d0f42000940fea7"
  ],
  "unseal_shares": 5,
  "unseal_threshold": 3,
  "recovery_keys_b64": [],
  "recovery_keys_hex": [],
  "recovery_keys_shares": 0,
  "recovery_keys_threshold": 0,
  "root_token": "hvs.sox6HrUL6hZOP0vM3lCszVQj"
}

First of all, we need to create a specific token with limited permissions based on our root token to unseal Vault by itself.

kubectl -n vault get pods
# NAME                       READY   STATUS    RESTARTS        AGE
# vault-0                    0/1     Running   0               6d18h
# vault-csi-provider-jw7jc   2/2     Running   0               6d18h

As we can see, Vault is sealed, so we need to unseal it manually using the unseal-keys.

kubectl -n vault exec -it vault-0 -- sh
/ $ vault operator unseal JDM9anYtanlo5iNYJfmCCIa5jsGJjVaKxYBdn98YQYmJ
/ $ vault operator unseal Nk1W+oDisOL7G6ad88ULnV0ZHRahTqe5as76LjbI43s0
/ $ vault operator unseal IKAfWjbA2l41mb4JwPJdSl+7NcM3ZwHpMr2JvyCpZTdl
# Key                     Value
# ---                     -----
# Seal Type               shamir
# Initialized             true
# Sealed                  false
# Total Shares            5
# Threshold               3/5
# Version                 1.18.1
# Build Date              2024-10-29T14:21:31Z
# Storage Type            raft
# Cluster Name            vault-cluster-9763a41e
# Cluster ID              23b3d5d7-9552-7f99-7330-45e52af54b78
# HA Enabled              true
# HA Cluster              n/a
# HA Mode                 standby
# Active Node Address     <none>
# Raft Committed Index    2706
# Raft Applied Index      2706

/ $ vault login
# Token (will be hidden): <root_token> 
# Success! You are now authenticated. The token information displayed below
# is already stored in the token helper. You do NOT need to run "vault login"
# again. Future Vault requests will automatically use this token.

# Key                  Value
# ---                  -----
# token                hvs.sox6HrUL6hZOP0vM3lCszVQj
# token_accessor       EQm9yoRLbCrvtuEUjInEVnAp
# token_duration       ∞
# token_renewable      false
# token_policies       ["root"]
# identity_policies    []
# policies             ["root"]

Now, Vault is unseal, let's create a policy with specific capabilities.

vault policy write unseal-policy - <<EOF
    path "sys/seal-status" {
        capabilities = ["read"]
    }

    path "sys/unseal" {
        capabilities = ["read", "update"]
    }
EOF
# Success! Uploaded policy: unseal-policy

We create a token to unseal Vault as per above policy with no expiration.

vault token create -policy="unseal-policy" -ttl="0"
# Key                  Value
# ---                  -----
# token                hvs.CAESIOD0f4NE5X-nu2wUdSbr-Z4-zz5_vuwSI4cs7vUmp1QgGh4KHGh2cy5Ed0xTNXZRRjZXYzQ0Q1A2TlRma0taNHU
# token_accessor       MKLIU30GAU0oaenqJKdzoCM9
# token_duration       768h
# token_renewable      true
# token_policies       ["default" "unseal-policy"]
# identity_policies    []
# policies             ["default" "unseal-policy"]

In this case the token is hvs.CAESIOD0f4NE5X-nu2wUdSbr-Z4-zz5_vuwSI4cs7vUmp1QgGh4KHGh2cy5Ed0xTNXZRRjZXYzQ0Q1A2TlRma0taNHU.

Let's configure an extra container in charge to unseal Vault every time it is initialized.

For that, we need to create a secret file that contains the unseal keys and the token with specific capabilities.

We define the secrets in base64 and deploy them as follows.

apiVersion: v1
kind: Secret
metadata:
  name: vault-secrets-unseal
  namespace: vault
data:
  VAULT_TOKEN: aHZzLkNBRVNJT0QwZjRORTVYLW51MndVZFNici1aNC16ejVfdnV3U0k0Y3M3dlVtcDFRZ0doNEtIR2gyY3k1RWQweFROWFpSUmpaWFl6UTBRMUEyVGxSbWEwdGFOSFUK
  UNSEAL_TOKEN1: SkRNOWFuWXRhbmxvNWlOWUpmbUNDSWE1anNHSmpWYUt4WUJkbjk4WVFZbUoK
  UNSEAL_TOKEN2: SUtBZldqYkEybDQxbWI0SndQSmRTbCs3TmNNM1p3SHBNcjJKdnlDcFpUZGwK
  UNSEAL_TOKEN3: TmsxVytvRGlzT0w3RzZhZDg4VUxuVjBaSFJhaFRxZTVhczc2TGpiSTQzczAK
type: Opaque

kubectl apply -f vault-secrets-unseal.yaml

Now, we need to get the values from Vault, as follows.

helm get values vault -n vault -o yaml > values.yaml

The values will look something like this.

csi:
  enabled: true
injector:
  enabled: false
server:
  ha:
    enabled: false
    raft:
      enabled: true
      storage:
        size: 10Gi
        storageClass: standard
.
.
.

So, we need to edit it, adding an extra container in charge to execute the unsealing process. The values should be something like this.

csi:
  enabled: true
injector:
  enabled: false
server:
  ha:
    enabled: false
    raft:
      enabled: true
      storage:
        size: 10Gi
        storageClass: standard
  securityContext:
    runAsUser: 0 
    privileged: true
  
  ## Add this part
  extraContainers:
    - name: vault-auto-unseal
      image: alpine-with-jq-curl
      imagePullPolicy: IfNotPresent
      command:
        - "/bin/sh"
        - "-c"
        - |
          until curl -s --fail $VAULT_URL/v1/sys/seal-status; do
            echo "Waiting for Vault to be ready..."
            sleep 5
          done

          # Get Vault's state
          vaultStatus=$(curl -s --request GET $VAULT_URL/v1/sys/seal-status | jq '.sealed')

          if [ "$vaultStatus" = "true" ]; then
            
            echo "Vault is sealed. Starting unseal process..."

            # Delete spaces
            vault_token=$(echo "$VAULT_TOKEN" | tr -d '[:space:]')

            for token_var in UNSEAL_TOKEN1 UNSEAL_TOKEN2 UNSEAL_TOKEN3; do
              unseal_token=$(eval echo \$$token_var)

              # Create payload
              payload=$(jq -n --arg key "$unseal_token" '{key: $key}') 

              curl --header "X-Vault-Token: $vault_token" \
                  --request PUT \
                  --data "$payload" \
                  $VAULT_URL/v1/sys/unseal
            done
            
            echo "Vault has been unsealed successfully."
          
          else
            echo "Vault is unseal."
          fi

          tail -f /dev/null
      env:
        - name: VAULT_URL
          value: "http://vault.vault.svc.cluster.local:8200"
        - name: VAULT_TOKEN
          valueFrom:
            secretKeyRef:
              name: vault-secrets-unseal  
              key: VAULT_TOKEN
        - name: UNSEAL_TOKEN1
          valueFrom:
            secretKeyRef:
              name: vault-secrets-unseal  
              key: UNSEAL_TOKEN1
        - name: UNSEAL_TOKEN2
          valueFrom:
            secretKeyRef:
              name: vault-secrets-unseal  
              key: UNSEAL_TOKEN2
        - name: UNSEAL_TOKEN3
          valueFrom:
            secretKeyRef:
              name: vault-secrets-unseal  
              key: UNSEAL_TOKEN3

NOTE: As you could see, I recommed using an lightweight image like alpine with jq and curl installed. You can install it only doing apk add curl jq, or build a personalized image like the one below.

FROM alpine:latest

RUN apk update && apk add jq curl

Apply the changes.

helm upgrade --install vault hashicorp/vault --values vault-values.yaml -n vault

Check Vault behavior.

kubectl -n vault get pods
# NAME                       READY   STATUS    RESTARTS   AGE
# vault-0                    2/2     Running   0          46m
# vault-csi-provider-frm9z   2/2     Running   0          100m

kubectl -n vault logs vault-0 -c vault-auto-unseal
# Waiting for Vault to be ready...
# Waiting for Vault to be ready...
# Waiting for Vault to be ready...
# Vault is sealed. Starting unseal process...
# Vault has been unsealed successfully.
# Vault is unseal.

Additionally, we can verify the expected behavior by deleting the Vault pod.

kubectl -n vault delete pod vault-0
kubectl -n vault get pods
# NAME                       READY   STATUS    RESTARTS   AGE
# vault-0                    2/2     Running   0          6s
# vault-csi-provider-frm9z   2/2     Running   0          108m

Conclusion

Implementing an auto-unseal mechanism for Vault without relying on a cloud-based KMS provides greater flexibility and control over your security infrastructure. This approach ensures that Vault remains operational even in environments with limited or no internet access, reducing external dependencies that might introduce security risks or downtime due to cloud provider issues.

autounseal vault - juancamilocc/virtual_resources GitHub Wiki

Auto-unseal Vault in Kubernetes environments

Conclusion

⚠️ GitHub.com Fallback ⚠️

autounseal vault - juancamilocc/virtual_resources GitHub Wiki

Auto-unseal Vault in Kubernetes environments

Conclusion

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️