vm broker ‐ k8s ‐ tenants ‐ kes vault (no TLS) - allanrogerr/public GitHub Wiki
Run a minio tenant with kes/kvm (hashicorp vault)
Prerequisites - setup operator first. See https://github.com/allanrogerr/public/wiki/vm-broker-%E2%80%90-k8s-%E2%80%90-k3s-%E2%80%90-operator
Vault setup
On the k8s enabled vm broker instance, https://kes-k8s-minio.lab.min.dev/, deploy a non-TLS Hashicorp vault pod
ssh -p 20050 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
loginctl enable-linger ubuntu
mkdir ~/github/operator && cd ~/github && git clone https://github.com/minio/operator.git && cd ~
kubectl apply -f ~/github/operator/examples/vault/deployment.yaml
Expose vault with a k8s port-forward
kubectl port-forward svc/vault 8200 &
Get general information on vault pod
kubectl logs -l app=vault
Output
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.
You may need to set the following environment variables:
$ export VAULT_ADDR='http://0.0.0.0:8200'
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: KlD6iy45uXgsEPQqRmSU1Iz6BunB3EUbmJpHy1RU32o=
Root Token: hvs.GlN4J7anzB8YsKIfrChxIIN5
Development mode should NOT be used in production installations!
Setup vault pod - inspired by https://github.com/minio/operator/blob/master/docs/examples.md
Access
kubectl -n default get pods
Output
NAME READY STATUS RESTARTS AGE
vault-544b87497f-qqtdr 1/1 Running 0 40s
kubectl exec -it vault-544b87497f-qqtdr -n default -- /bin/sh
Setup
/ #
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=hvs.GlN4J7anzB8YsKIfrChxIIN5
vault auth enable approle
vault secrets enable -version=1 kv
vault secrets list
vi kes-policy.hcl
###
path "kv/*" {
capabilities = [ "create", "read", "update", "patch", "delete", "list" ]
}
###
vault policy write kes-policy kes-policy.hcl
vault write auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m policies=kes-policy
vault read auth/approle/role/kes-role/role-id
###
Key Value
--- -----
role_id 99366a8e-b81b-b7b5-fc50-b3edacf674f9
###
vault write -f auth/approle/role/kes-role/secret-id
###
Key Value
--- -----
secret_id c82e62b8-bbb0-957a-2cdd-a7af9b4cd29c
secret_id_accessor 735050b5-1276-17c1-412d-1cafc44cfe82
secret_id_num_uses 0
secret_id_ttl 0s
###
exit
Alter config.yml to use the non TLS enabled vault
vi ~/github/operator/examples/kustomization/tenant-kes-encryption/kes-configuration-secret-demo.yaml
apiVersion: v1
kind: Secret
metadata:
name: kes-configuration
namespace: tenant-kms-encrypted
type: Opaque
stringData:
server-config.yaml: |-
version: v1
address: :7373
admin:
identity: ${MINIO_KES_IDENTITY}
tls:
key: /tmp/kes/server.key
cert: /tmp/kes/server.crt
proxy:
identities: []
header:
cert: X-Tls-Client-Cert
policy:
my-policy:
allow:
- /v1/api
- /v1/key/create/*
- /v1/key/generate/*
- /v1/key/decrypt/*
- /v1/key/bulk/decrypt/*
- /v1/key/list/*
- /v1/status
identities:
cache:
expiry:
any: 5m0s
unused: 20s
log:
error: on
audit: off
keystore:
vault:
version: "v1"
endpoint: "http://vault.default.svc.cluster.local:8200"
namespace: ""
prefix: "my-minio"
approle:
id: "99366a8e-b81b-b7b5-fc50-b3edacf674f9"
secret: "c82e62b8-bbb0-957a-2cdd-a7af9b4cd29c"
retry: 15s
status:
ping: 10s
Alter the kustomize command to use the new config.yaml
vi ~/github/operator/examples/kustomization/tenant-kes-encryption/kustomization.yaml
###
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: tenant-kms-encrypted
resources:
- ../base
- kes-configuration-secret-demo.yaml
patchesStrategicMerge:
- tenant.yaml
###
Install - kustomize
sudo snap install kustomize
kustomize build ~/github/operator/examples/kustomization/tenant-kes-encryption | kubectl apply -f -
Tweak tenant size and storeclass/pvcs
kubectl patch tenant -n tenant-kms-encrypted myminio --type='merge' -p '{"spec":{"pools":[{"name": "pool-0", "servers": 4, "volumesPerServer": 1, "volumeClaimTemplate": {"apiVersion": "v1", "metadata": {"name": "data"}, "spec": {"accessModes": ["ReadWriteOnce"], "resources": {"requests": {"storage": "1Gi"}}, "storageClassName": "local-path"}}}]}}'
kubectl -n tenant-kms-encrypted delete statefulset/myminio-pool-0
kubectl -n tenant-kms-encrypted delete statefulset/myminio-kes
Add a quick validating port forward
kubectl port-forward svc/minio 9000:443 -n tenant-kms-encrypted &
mkdir ~/mc && cd ~/mc && wget https://dl.min.io/client/mc/release/linux-amd64/mc && cd ~
chmod +x mc
mc/mc alias set alias https://127.0.0.1:9000 minio minio123 --insecure
mc/mc admin kms key status alias --insecure
Output
Handling connection for 9000
Key: my-minio-key
- Encryption ✔
- Decryption ✔
Test on minio - get jwt
SA_TOKEN=$(kubectl -n minio-operator get secret console-sa-secret -o jsonpath="{.data.token}" | base64 --decode)
echo $SA_TOKEN
eyJhbGciOiJSUzI1NiIsImtpZCI6ImY1NTlPM3M5d2d0QU5HYURTTXd2V0pHTHBHMUFjTXZDUVNjQ2s4MFNndlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjb25zb2xlLXNhLXNlY3JldCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjb25zb2xlLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMDQ4NDAwYmQtMDExNi00YTIyLTljNTItMDJhMmQzNjE4NmI2Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om1pbmlvLW9wZXJhdG9yOmNvbnNvbGUtc2EifQ.e38RvwqANpjaovvCzPy7y_ZfzpY1SoJWEBZJqr9Q41oGtFfHyA5FGvKLYV6O9zhP26W1k1DOkbmhjlGBy3Kc2GKYTU7HbGmPZwUagw2D1jMQwlF6GecM6dusif29Zmdf8uz0Dt7cIn8d5sKPLLlJY_75amRhx7tMUylCDO-Q4YyA2bC3WZIv1j89UzN3e2zeCAzC0v4eA6PqArrxxWMCNQlDXHp_Cv29eFLeojQbxGrahpVt0P14AodOBX1SuUIuvJDofqr9ttcSe1pim7CjwOkn4lKDtqjLiuoWSvIzlNI1T0zQ-ftGwxCOx038cRM-ROV0Jg25Clo0dYz7492uQQ
Create a NodePort and access the operator/tenant at https://kes-k8s-minio.lab.min.dev:30043
kubectl patch service -n minio-operator console -p '{"spec":{"ports":[{"name": "http","port": 9090,"protocol": "TCP","nodePort":31090}],"type": "NodePort"}}'
kubectl patch service -n minio-operator console -p '{"spec":{"ports":[{"name": "https","port": 9443,"protocol": "TCP","nodePort":30043}],"type": "NodePort"}}'
Corollary - Use API key to connect to minio to kes, instead of a cert/key
Connect to kes pod, create new identity
kubectl exec -it myminio-kes-0 -n tenant-kms-encrypted -- /bin/sh
./kes identity new
exit
Output
Your API key:
kes:v1:ABEeVWZshnKsxaUx+H3lSYCOKZDh63AazG1rhQGnG3BQ
This is the only time it is shown. Keep it secret and secure!
Your Identity:
c5eb01c9a4f9c4ef2fba7653cd98ff4d928bc48ec71ad7356fd39711fbed74b9
The identity is not a secret. It can be shared. Any peer
needs this identity in order to verify your API key.
The identity can be computed again via:
kes identity of kes:v1:ABEeVWZshnKsxaUx+H3lSYCOKZDh63AazG1rhQGnG3BQ
Add identity as admin of kes config.yaml
vi ~/github/operator/examples/kustomization/tenant-kes-encryption/kes-configuration-secret-demo.yaml
Change .admin.identity to c5eb01c9a4f9c4ef2fba7653cd98ff4d928bc48ec71ad7356fd39711fbed74b9
Comment the following to demonstrate the API key is being used .policy.my-policy.allow.identities.${MINIO_KES_IDENTITY}
Recreate tenant and kes pods
kustomize build ~/github/operator/examples/kustomization/tenant-kes-encryption | kubectl apply -f -
kubectl -n tenant-kms-encrypted delete statefulset/myminio-pool-0
kubectl -n tenant-kms-encrypted delete statefulset/myminio-kes
In tenant UI https://kes-k8s-minio.lab.min.dev:30043/, add the following. Save and restart the tenant
MINIO_KMS_KES_API_KEY="kes:v1:ABEeVWZshnKsxaUx+H3lSYCOKZDh63AazG1rhQGnG3BQ"
MINIO_KMS_KES_CERT_FILE=""
MINIO_KMS_KES_KEY_FILE=""
kubectl patch tenant -n tenant-kms-encrypted myminio --type='merge' -p '{"spec":{"pools":[{"name": "pool-0", "servers": 4, "volumesPerServer": 1, "volumeClaimTemplate": {"apiVersion": "v1", "metadata": {"name": "data"}, "spec": {"accessModes": ["ReadWriteOnce"], "resources": {"requests": {"storage": "1Gi"}}, "storageClassName": "local-path"}}}]}}'
kubectl -n tenant-kms-encrypted delete statefulset/myminio-pool-0
kubectl -n tenant-kms-encrypted delete statefulset/myminio-kes
Observe the variables are set and that the minio pods are Running
kubectl -n tenant-kms-encrypted get secret/storage-configuration -o json | jq '.data."config.env"' | base64 -di
Output
export MINIO_ROOT_USER="minio"
export MINIO_KMS_KES_API_KEY="kes:v1:ABEeVWZshnKsxaUx+H3lSYCOKZDh63AazG1rhQGnG3BQ"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_KMS_KES_CERT_FILE=""
export MINIO_STORAGE_CLASS_STANDARD="EC:2"
export MINIO_BROWSER="on"
export MINIO_KMS_KES_KEY_FILE=""
Create a bucket, KMS Key and successfully upload data using encryption
Test client access
Get kes client secrets
kubectl -n tenant-kms-encrypted get secret/myminio-client-tls -o yaml
Within kes pod - convert to certs and use to authenticate to vault
kubectl exec -it myminio-kes-0 -n tenant-kms-encrypted -- /bin/sh
echo "LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1DNENBUUF3QlFZREsyVndCQ0lFSUlITXpIeXRlNC8xbWVNdE5HRzJLZHh1VTFnV1V5NEFEUzYrOVJKZ3V0emUKLS0tLS1FTkQgUFJJVkFURSBLRVktLS0tLQo=" | base64 -d> /tmp/private.key
echo "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJ3akNDQVhTZ0F3SUJBZ0lRRk9QQmYwTy9yQWMvL1BwNkNZMUtwVEFGQmdNclpYQXdOekUxTURNR0ExVUUKQXhNc2JXbHVhVzh1ZEdWdVlXNTBMV3R0Y3kxbGJtTnllWEIwWldRdWMzWmpMbU5zZFhOMFpYSXViRzlqWVd3dwpIaGNOTWpNeE1URXpNakExTnpRNVdoY05NalF4TVRFeU1qQTFOelE1V2pBM01UVXdNd1lEVlFRREV5eHRhVzVwCmJ5NTBaVzVoYm5RdGEyMXpMV1Z1WTNKNWNIUmxaQzV6ZG1NdVkyeDFjM1JsY2k1c2IyTmhiREFxTUFVR0F5dGwKY0FNaEFQMnRwQW5FVmxUYkNoOGNkaEJRaGlReVBIQ2FKZTNkQ0dqZ25lN0FCRWswbzRHVk1JR1NNQTRHQTFVZApEd0VCL3dRRUF3SUhnREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQVFZSUt3WUJCUVVIQXdJd0RBWURWUjBUCkFRSC9CQUl3QURCVEJnTlZIUkVFVERCS2draHRlVzFwYm1sdkxYQnZiMnd0TUMxN01DNHVMak45TG0xNWJXbHUKYVc4dGFHd3VkR1Z1WVc1MExXdHRjeTFsYm1OeWVYQjBaV1F1YzNaakxtTnNkWE4wWlhJdWJHOWpZV3d3QlFZRApLMlZ3QTBFQTBmNi9kREpZN3lSNXNBT0xIMzB5NWFyV25jMXdhNTNBbWRoNm9ZbnYxWW95clZtVFVrRjJQdDZCClN6UWR5UWR2VTRWdTBhK1pBREUyOFlmQjlISUpDdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K" | base64 -d > /tmp/public.crt
export KES_SERVER=https://127.0.0.1:7373
export KES_CLIENT_CERT=/tmp/public.crt
export KES_CLIENT_KEY=/tmp/private.key
./kes key ls -k
Key
my-minio-key
Test/debug direct vault functionality
Using vault admin token
kubectl exec -it vault-544b87497f-rnk57 -n default -- /bin/sh
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=hvs.FlUxgPci96dRsZ1MM3wGmsKu
vault status
Outside of vault pod, test login token generated
curl \
--request POST \
--data '{"role_id":"ad119367-50f4-30a3-3e1b-67be2297d9d4","secret_id":"600d8ddb-d7a4-e0e4-def4-ecf44094da8c"}' \
http://127.0.0.1:8200/v1/auth/approle/login | jq .
Output
{
"request_id": "a12ac7ae-d050-7027-80a1-01b4abd334ca",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": null,
"wrap_info": null,
"warnings": null,
"auth": {
"client_token": "hvs.CAESIAXAJGvHK7ax055D7Nc_4adMECwXP3lNEikwpVY3NfdxGh4KHGh2cy44RE55S3MxbXV2dUdLVldxWHpkTEl2dGM",
"accessor": "aYUDxtuuwXPIKv3fTswAW0U0",
"policies": [
"default",
"kes-policy"
],
"token_policies": [
"default",
"kes-policy"
],
"metadata": {
"role_name": "kes-role"
},
"lease_duration": 300,
"renewable": true,
"entity_id": "55abbdb5-7cf7-47f5-3b4d-52cde703cfd4",
"token_type": "service",
"orphan": true,
"mfa_requirement": null,
"num_uses": 0
}
}
Test login token within vault pod
kubectl exec -it vault-544b87497f-rnk57 -n default -- /bin/sh
export VAULT_ADDR=http://localhost:8200
export USE_LIMIT_TOKEN="hvs.CAESIAXAJGvHK7ax055D7Nc_4adMECwXP3lNEikwpVY3NfdxGh4KHGh2cy44RE55S3MxbXV2dUdLVldxWHpkTEl2dGM"
VAULT_TOKEN=$USE_LIMIT_TOKEN vault token lookup
###
Key Value
--- -----
accessor aYUDxtuuwXPIKv3fTswAW0U0
creation_time 1699927933
creation_ttl 5m
display_name approle
entity_id 55abbdb5-7cf7-47f5-3b4d-52cde703cfd4
expire_time 2023-11-14T02:17:13.313631843Z
explicit_max_ttl 0s
id hvs.CAESIAXAJGvHK7ax055D7Nc_4adMECwXP3lNEikwpVY3NfdxGh4KHGh2cy44RE55S3MxbXV2dUdLVldxWHpkTEl2dGM
issue_time 2023-11-14T02:12:13.313637564Z
meta map[role_name:kes-role]
num_uses 0
orphan true
path auth/approle/login
period 5m
policies [default kes-policy]
renewable true
ttl 4m7s
type service
###
Generate service token for API/CLI based on kes-policy within vault pod
kubectl exec -it vault-544b87497f-rnk57 -n default -- /bin/sh
vault token create -ttl=1h -use-limit=20 -policy=kes-policy
Test service token outside of vault pod
Create a secret
export VAULT_TOKEN="hvs.CAESIJk2tgpboDe3fEdQ_FWpITNXiBf-7fsGHGjPsM3rKD2jGh4KHGh2cy5TVDZRNzl4eU9TNWR1ZkdYWjBkRWw0MWQ"
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--header "Content-Type: application/json" \
--request POST \
--data '{ "data": {"password_kes": "my-short-passwords"} }' \
http://127.0.0.1:8200/v1/kv/creds_kes
Read a secret
export VAULT_TOKEN="hvs.CAESIAXAJGvHK7ax055D7Nc_4adMECwXP3lNEikwpVY3NfdxGh4KHGh2cy44RE55S3MxbXV2dUdLVldxWHpkTEl2dGM"
curl \
-H "X-Vault-Token: $VAULT_TOKEN" \
-X GET \
http://127.0.0.1:8200/v1/kv/creds_kes | jq
Output
{
"request_id": "df964d55-9305-340f-2fc0-8c99a4c98fb9",
"lease_id": "",
"renewable": false,
"lease_duration": 2764800,
"data": {
"data": {
"password_kes": "my-short-passwords"
}
},
"wrap_info": null,
"warnings": null,
"auth": null
}
List
curl \
-H "X-Vault-Token: $VAULT_TOKEN" \
-X LIST \
http://127.0.0.1:8200/v1/kv | jq
Output
{
"request_id": "1692c256-79bc-ae50-c884-346b7da283ce",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": {
"keys": [
"creds_kes",
"my-minio/"
]
},
"wrap_info": null,
"warnings": null,
"auth": null
}
Vault debugging
Which policies actually exist? Is these a kes-policy?
Read policies
kubectl exec -it vault-544b87497f-rnk57 -n default -- /bin/sh
export VAULT_ADDR=http://localhost:8200
vault read sys/policy
Output
Key Value
--- -----
keys [default kes-policy root]
policies [default kes-policy root]
vault policy list
default
kes-policy
root
What capabilities does this kes-policy possess?
vault policy read kes-policy
path "kv/*" {
capabilities = [ "create", "read", "delete" ]
}
Will a token with this policy and thus capabilities allow me to do xyz?
vault token create -format=json -policy="kes-policy"
{
"request_id": "484bd87c-95ac-aa98-6a8b-ebe21a5f0bc7",
"lease_id": "",
"lease_duration": 0,
"renewable": false,
"data": null,
"warnings": null,
"auth": {
"client_token": "hvs.CAESIO_88xMhB-WW19DF8rygfhmsV29uOwTC5svhWrQ3VwzbGh4KHGh2cy41NlFnc09nbGxZSDVJR3dpTUtwVlZaVWY",
"accessor": "eqfLNQawfleGWKOAGxqc4UgE",
"policies": [
"default",
"kes-policy"
],
"token_policies": [
"default",
"kes-policy"
],
"identity_policies": null,
"metadata": null,
"orphan": false,
"entity_id": "",
"lease_duration": 2764800,
"renewable": true,
"mfa_requirement": null
}
}
KES_TOKEN="hvs.CAESIO1UN1hwZym-IgBXscJhySK2I3wEKPfmZScGydM8OotPGh4KHGh2cy5LaDJrbXZRSTJOcVVzeGh1VlF1dmFwSm0"
vault token capabilities $KES_TOKEN kv/*
create, delete, read
Use this token to Test service token outside of vault pod (see above)