Verifying Segment Store TLS Reload in Kubernetes clusters - derekm/pravega GitHub Wiki

Overview

It's easy to verify TLS reload using Pravega standalone - just replace the segment store certificates with new ones. That feature was built as part of pull request 3966 - Segment store TLS reload upon certificate refresh

How do you verify TLS reload in Kubernetes (K8S) clusters? Since we use K8S secrets to deploy the certificates in these clusters, you can't just replace the files or change their timestamps using the touch command. The steps below explain how to do it.

• Overview
• Setting up a Pravega cluster for dynamic TLS Certificate reloading
  • Step 1: Deploy Cert Manager
  • Step 2: Setup the Certificate Issuer
  • Step 3: Deploy Pravega with Appropriate Config

Setting up a Pravega cluster for dynamic TLS Certificate reloading

Step 1: Deploy Cert Manager

Note: Here, we specify v0.9 of the Cert Manager, which was used at the time of development of the pull request. You might want to use the latest version, instead.

# Install the cert-manager CRDs. We must do this before installing the Helm
# chart in the next step for `release-0.9` of cert-manager:
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.9/deploy/manifests/00-crds.yaml

# Create the namespace for cert-manager
kubectl create namespace cert-manager

# Label the cert-manager namespace to disable resource validation
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

## Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io

## Updating the repo just incase it already existed
helm repo update

## Install the cert-manager helm chart
helm install \
  --name cert-manager \
  --namespace cert-manager \
  --version v0.9.0 \
  jetstack/cert-manager

Step 2: Setup the Certificate Issuer

References:

• https://cert-manager.io/docs/configuration/ca/

Here, we'll use an existing CA as the issuer in Cert Manager.

1. Create a secret using the CA certificate and key.

   kubectl create secret tls ca-key-pair `
       --cert=ca.crt \
       --key=ca.key \
       --namespace=default
   

2. Verify that the secret is created properly.

   kubectl get secret ca-key-pair
   kubectl describe secret ca-key-pair
   

3. Create an issuer configuration file (issuer.yaml file) using the ca-key-pair secret.

   apiVersion: certmanager.k8s.io/v1alpha1
   kind: Issuer
   metadata:
     name: ca-issuer
     namespace: default
   spec:
     ca:
       secretName: ca-key-pair 
   

Step 3: Deploy Pravega with Appropriate Config

1. Follow the usual steps until you reach the point where you need to create Controller secret with TLS material and auth related files.
2. Create Controller secret as usual.

kubectl create secret generic controller-tls \
   --from-file=./controller01.pem \
   --from-file=./ca.crt \
   --from-file=./controller01.key.pem \
   --from-file=./controller01.jks \
   --from-file=./pass-secret-tls

3. Create a dummy Segment Store secret (since 0.8.0), although the certs and keys will really be injected by Cert Manager (see below steps)

kubectl create secret generic segmentstore-tls \
   --from-file=./segmentstore01.pem \
   --from-file=./ca-cert \
   --from-file=./segmentstore01.key.pem 

4. Create a Cert Manager secret creation yaml manifest segmentstore-cert-request.yaml. Note:
   • Use the appropriate secret name (segmentstore-tls in the example below). It has to match what you are going to use in your K8S deployment manifest.
   • Adjust the duration and renewBefore according to your needs. They determine how often the certificate is refreshed. See https://docs.cert-manager.io/en/release-0.9/reference/certificates.html for documentation. duration specifies the certificate validity duration. renewBefore specifies the amount of time before expiry when the certificate will be renewed. So, if the two are 1 hour and 45 minutes respectively, the certificate will get renewed approximately in 15 minutes after the first issue.
   • Adjust the common names and dns names according to your needs. They determine the contents of the CN and SAN fields, respectively.

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: segmentstore-cert
  namespace: default
spec:
  secretName: segmentstore-tls
  keyEncoding: pkcs8
  duration: 1h
  renewBefore: 45m
  issuerRef:
    name: ca-issuer
    kind: Issuer
  commonName: segmentstore.pravega.io
  organization:
  - Example CA
  dnsNames:
  - segmentstore.pravega.io
  - segmentstore2.pravega.io

5. Setup segment store automatic certificate distribution for segment store:

kubectl create -f segmentstore-cert-request.yaml

#Verify
kubectl describe secret segmentstore-tls
kubectl get secret segmentstore-tls -o yaml
kubectl get secrets

6. Deploy Pravega as usual with TLS and Auth enabled and this additional param shown below. Also, enable debug-level logging

# New config property (since 0.8.x)
pravegaservice.security.tls.certificate.autoReload.enable

# or, alternatively old config property (for older versions than 0.8.x)
pravegaservice.enableTlsReload=true 

7. Wait for the cluster to get deployed. Verify that you are able to access the cluster. Then, wait for an appropriate duration based on the duration and renewBefore set earlier for the certificate to get reloaded. Once the certificate is reloaded, you should observe statements that indicate certificate reload in segment store logs.

2020-07-08 11:08:06,937 801732 [Thread-2] INFO  i.p.s.s.h.s.TLSConfigChangeHandler - Current reload count = 1
2020-07-08 11:22:36,998 1671793 [Thread-2] INFO  i.p.s.s.h.s.TLSConfigChangeHandler - Current reload count = 2
2020-07-08 11:37:47,054 2581849 [Thread-2] INFO  i.p.s.s.h.s.TLSConfigChangeHandler - Current reload count = 3
2020-07-08 11:52:47,105 3481900 [Thread-2] INFO  i.p.s.s.h.s.TLSConfigChangeHandler - Current reload count = 4