Configuring Hashicorp Vault Instance for storing Secrets - vmware/versatile-data-kit GitHub Wiki

• Overview
• Prerequisites
• AppRole Configuration in Hashicorp Vault
• VDK Control Service Configuration
• Conclusion

The VDK Control Service allows for the optional configuration of a Hashicorp Vault instance for storing secrets. This allows data jobs users to store sensitive data encrypted in Vault, rather than using properties that are stored in plain text in the VDK Control Service database.

As part of this tutorial, you will learn how to configure the VDK Control Service to use Hashicorp Vault to store Secrets. This is a two-step process:

• Configuring AppRoles in a Hashicorp vault and obtaining the required configuration
• Configuration changes in the VDK Control Service

This article is for VDK Control Service Infrastructure Operators.

If you have running instances of the VDK Control Service and Hashicorp Vault, the completion of this tutorial should take 15 to 30 minutes.

1. A VDK Control Service installation Install VDK Control Service with custom SDK
2. A running instance of Hashicorp Vault

As a first step, you are going to configure an AppRole in your Vault instance. The AppRole auth method allows machines or apps to authenticate with Vault-defined roles. Please follow the official AppRroles guide to perform the following actions:

1. Enable AppRole authentication
2. Create a policy named "vdk-policy" which allows all operations on the secrets engine/path you want to use in Vault
3. Create an AppRole named "vdk-role" and assign the "vdk-policy" you created earlier
4. Obtain the the "vdk-role" RoleID
5. Obtain the the "vdk-role" SecretID

Once you have the Vault URL and the "vdk-role" RoleID and SecretID items, you are ready to make the configuration changes of the VDK Control Service.

At this point, you are going to configure the VDK Control service to use the AppRole you created in the previous section of this tutorial by adding/editing the "secrets" section of your VDK Control Service values.yaml file. Make the following changes:

# Vault integration configuration for storing Data Job Secrets
secrets:
    vault:
        enabled: true
        uri: "<vault URL>/v1/"
        approle:
            roleid: "<vdk-role RoleId>"
            secretid: "<vdk-role SecretId>"

NOTE: in most cases, you should append "/v1/" to the end of the Vault URL for AppRole Authentication to work

Apply the changes to the configuration of your Control Service, by executing the following command:

helm upgrade --install my-vdk-runtime vdk-gitlab/pipelines-control-service -f values.yaml

Wait for the upgrade process to finish.

Congratulations! You have successfully configured the VDK Control Service / Hashicrop Vault integration in your environment and your data jobs users can store secrets securely.

Check out the tutorial on Ingesting data from a authenticated REST API using Job Secrets where you can learn how to use secrets in practice.