PVC Migration - bcgov/common-service-showcase GitHub Wiki

Persistent Volume Claim Migration

There may be times when the contents in a Persistent Volume Claim need to be migrated to a different PVC. Some potential reasons for this include:

  • More PVC storage space is required than is currently allocated
  • The storage class needs to change for some reason
  • The existing PVC no longer satisfies requirements

Table of Contents

High Level Strategy

A PVC migration, regardless of the contents inside the PVC, will generally follow this general strategy:

  1. Add any temporary permissive NSP rules to namespace
  2. Shut down the service(s) which use the PVC (may not be needed if it is Highly Available)
  3. Create a new temporary PVC to hold the data
  4. Create a temporary pod which mounts both the source and temporary PVCs
  5. Copy the contents of the source to the temporary
  6. Kill the temporary pod and the source PVC
  7. Regenerate the source PVC with the desired parameters
  8. Create another temporary pod which mounts both the source and temporary PVCs
  9. Copy the contents of the temporary PVC back to the "source" PVC
  10. Kill the temporary pod and the temporary PVC
  11. Restore the service(s) which use the source PVC
  12. Remove any temporary NSP rules in namespace

In a nutshell, the process is to create a temporary PVC, graft the contents over to it, delete and recreate the PVC in question, and then graft the contents back. This pattern will generally apply to most PVC deployments as the contents are usually relatively mounted into the containers on startup.

Patroni Example

Patroni PVC migrations also follow the same high-level principles. However, as it is a self-healing, High Availability implementation, we can skip a few explicit steps and defer to Patroni to manage the copying for us instead. For more details on how this could be achieved, refer to the following Github Gist:

https://gist.github.com/jujaga/51c11383d07bf30f006e1610327c745c

Redis Example

Following the above high level strategy, these were the general commands and steps used to achieve the PVC migration with Redis. These commands can be reused with minor modifications to match other applications.

Add any temporary permissive NSP rules to namespace

The following adds generic NSP rules to allow all internal network traffic in the namespace to work.

export NAMESPACE=<YOURNAMESPACE>

oc process -n $NAMESPACE -f https://raw.githubusercontent.com/wiki/bcgov/common-service-showcase/assets/nsp.yaml -p NAMESPACE=$NAMESPACE -o yaml | oc apply -n $NAMESPACE -f -

Shut down the service(s) which use the PVC (may not be needed if it is Highly Available)

Before shutting down your service, it may be prudent to have some kind of metric to ensure that your contents are correct. In the case of Redis, we can note the number of unique keys stored in the DB with the following (done in pod):

redis-cli -h 127.0.0.1 -a $REDIS_PASSWORD info keyspace

We then shut down the service:

oc scale -n $NAMESPACE dc/<YOURAPPNAME> --replicas=0

Create a new temporary PVC to hold the data

This is an example PVC template called pvctemp.yaml. This was derived from the original Redis deployment template. You will want to adapt this to your application needs as necessary.

---
apiVersion: v1
kind: Template
labels:
  app: "${APP_NAME}-${JOB_NAME}"
  template: "${REPO_NAME}-template"
metadata:
  name: "${REPO_NAME}-redis-dc"
objects:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: redis-${JOB_NAME}
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: ${REDIS_PERSISTENT_VOLUME_CLASS}
      resources:
        requests:
          storage: "${REDIS_VOLUME_CAPACITY}"
parameters:
  - name: REPO_NAME
    description: Application repository name
    displayName: Repository Name
    required: true
    value: common-hosted-email-service
  - name: JOB_NAME
    description: Job identifier (i.e. 'pr-5' OR 'master')
    displayName: Job Branch Name
    required: true
    value: master
  - name: APP_NAME
    description: Application name
    displayName: Application name
    required: true
    value: ches
  - name: REDIS_VOLUME_CAPACITY
    description: Volume space available for Redis
    displayName: Redis Volume Capacity
    required: true
    value: 2Gi
  - name: REDIS_PERSISTENT_VOLUME_CLASS
    description: The storage class of the volume
    displayName: Persistent Volume Class name
    required: false
    value: netapp-file-standard

Given the above file, we then process and apply it:

oc process -n $NAMESPACE -f pvctemp.yaml -p JOB_NAME=master-temp -o yaml | oc apply -n $NAMESPACE -f -

Create a temporary pod which mounts both the source and temporary PVCs

There are many ways to create a new temporary pod, ranging from creating a new temporary deployment configuration with the PVCs mounted, or directly creating one through CLI. The following is an example using just the CLI to create a one-off pod named temp-pvc:

oc run -n $NAMESPACE temp-pvc --image=docker-registry.default.svc:5000/openshift/redis:latest --overrides='{"spec":{"containers":[{"command":["/bin/bash","-c","sleep 3600"],"image":"docker-registry.default.svc:5000/openshift/redis:latest","name":"temp-pvc","volumeMounts":[{"mountPath":"/source","name":"source"},{"mountPath":"/target","name":"target"}]}],"volumes":[{"name":"source","persistentVolumeClaim":{"claimName":"redis-master"}},{"name":"target","persistentVolumeClaim":{"claimName":"redis-master-temp"}}]}}' --restart=Never

Copy the contents of the source to the temporary

Remotely connect into the pod and then copy the contents. Keep a close eye on file permissions and ensure the files are not owned by root.

oc rsh -n $NAMESPACE temp-pvc
cp -a /source/. /target/

Kill the temporary pod and the source PVC

Delete the temp-pvc pod and then the source PVC (named redis-master).

oc delete -n $NAMESPACE pod temp-pvc
oc delete -n $NAMESPACE pvc redis-master

Regenerate the source PVC with the desired parameters

oc process -n $NAMESPACE -f pvctemp.yaml -p JOB_NAME=master -o yaml | oc apply -n $NAMESPACE -f -

Create another temporary pod which mounts both the source and temporary PVCs

Another one-off pod named temp-pvc is created. Note that the source and target PVC names have been inverted.

oc run -n $NAMESPACE temp-pvc --image=docker-registry.default.svc:5000/openshift/redis:latest --overrides='{"spec":{"containers":[{"command":["/bin/bash","-c","sleep 3600"],"image":"docker-registry.default.svc:5000/openshift/redis:latest","name":"temp-pvc","volumeMounts":[{"mountPath":"/source","name":"source"},{"mountPath":"/target","name":"target"}]}],"volumes":[{"name":"target","persistentVolumeClaim":{"claimName":"redis-master"}},{"name":"source","persistentVolumeClaim":{"claimName":"redis-master-temp"}}]}}' --restart=Never

Copy the contents of the temporary PVC back to the "source" PVC

Here you are effectively doing the reverse copy from earlier. Keep a close eye on file permissions and ensure the files are not owned by root.

oc rsh -n $NAMESPACE temp-pvc
cp /source/. /target/

Kill the temporary pod and the temporary PVC

Delete the temp-pvc pod and then the temporary PVC (named redis-master-temp).

oc delete -n $NAMESPACE pod temp-pvc
oc delete -n $NAMESPACE pvc redis-master-temp

Restore the service(s) which use the source PVC

Bring your service back online:

oc scale -n $NAMESPACE dc/<YOURAPPNAME> --replicas=1

Once the service is back up, you can double check the data integrity. In redis, you can run the following in the pod, and its count should match up with what you noted down earlier:

redis-cli -h 127.0.0.1 -a $REDIS_PASSWORD info keyspace

Remove any temporary NSP rules in namespace

Finally, remove any temporary NSP overrides you applied temporarily earlier. After this, your migration should be complete.

oc delete -n $NAMESPACE nsp -l template=networksecurity-template

Other Resources

Below are some potentially useful resources related to PVC migrations: