Expose vault with internal domain - johnzheng1975/devops_way GitHub Wiki

Expose vault with internal domain

Purpose

  • Need be securer when service get value from vault, even it is not using certificate.

Background

Generally, there are two ports for vault access, 8200 is http port, 8201 is https port for internal using. Client do not want to using certificate to access vault, since it is inconvience. However, using http 8200 is also not acceptable for security consideration. So, I tried to

  1. Put vault into istio mesh
  2. Enable mtls for vault 8200 port

Unfortunally, 8200 port is not only for accessing vault key/value, but also requried when injected pod start, or some unkonwn cases.

During my testing, even accessing key/value seems good, some error still raise in vault agent log. This may bring unpredictable risk.

Solution

Still put vault out of istio mesh. Only add an additional kubernetes service which provides https access.

You can copy service from vault service, and remove 8201 port.

Add additional service

This service is internal classic loadBalancer on aws

  • Using domain
  • Using AWS certificate manager for this domain
  • Only EKS IP range can access this loadBalancer
  • Forward 443 to 8200.
  • In AWS route53, add private hostname, then add record(cname) which connect domain and elb address.
apiVersion: v1
kind: Service
metadata:
  annotations:
    # external-dns.alpha.kubernetes.io/hostname: vault-dev-perf.xxx.io
    meta.helm.sh/release-name: vault
    meta.helm.sh/release-namespace: vault
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: Custom1=_cluster,Custom2=horizoncore-perf-dev-usw2-eks,Custom3=us-west-2
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-west-2:198814579854:certificate/xxxx-xxxx-xxxxx-b435-xxxxx
    service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS-1-2-2017-01 
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-internal: 10.106.128.0/18  
  labels:
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: vault
    helm.sh/chart: vault-0.8.0
  name: vault-external-2
  namespace: vault
spec:
  loadBalancerSourceRanges:
  - 10.106.128.0/18 
  ipFamilyPolicy: SingleStack
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8200
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/instance: vault
    app.kubernetes.io/name: vault
    component: server
  sessionAffinity: None
  type: LoadBalancer

Test result

npcore2@ip-172-20-223-79:~/tmp$ k exec -ti -n temp nginx -- bash
root@nginx:/# curl -H "X-Vault-Token: s.xxxxxxxxxxx"  -X GET -k https://vault-dev-perf.xxx.io/v1/hpbp-status/secrets -v
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 10.106.130.191:443...
* Connected to vault-dev-perf.xxx.io (10.106.130.191) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=California; L=Palo Alto; O=HP Inc; OU=PSG; CN=*.xxx.io
*  start date: Nov  8 00:00:00 2021 GMT
*  expire date: Nov  7 23:59:59 2022 GMT
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
> GET /v1/hpbp-status/secrets HTTP/1.1
> Host: vault-dev-perf.xxx.io
> User-Agent: curl/7.74.0
> Accept: */*
> X-Vault-Token: s.xxxxxxxxx
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Cache-Control: no-store
< Content-Type: application/json
< Date: Mon, 27 Dec 2021 04:06:28 GMT
< Content-Length: 585
< 
{"request_id":"a28aa86b-7fc2-f4ef-96b4-5a611e2f230b","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"awsAccessKey":"xxxxxx","awsAccessToken":"xxxxxxxxxxx","clientId":"f7f3cba4-b598-4a72-a7b1-5adb171c4680","clientSecret":"xxxx-xxxx","mongoUri":"mongodb+srv://hpbp-dev-:[email protected]/hpbp-dev-status?retryWrites=true","pagerDutyToken":"xxxx","redisToken":"xxxx"},"wrap_info":null,"warnings":null,"auth":null}
* Connection #0 to host vault-dev-perf.xxx.io left intact

Is this securer?

Answer: Yes, getting vault value is using https.

What if some client still using servicename.namespacename:8200 to get value？

Answer: We need highlight this rule to clients (applications). Besides this, we can use envoyfilter to block such access.