How to store sensitive information - anodot/daria GitHub Wiki

Data Collector pipeline stages communicate with sources to read and write data. Many of these sources require sensitive information, such as user names or passwords, to access the data. To save and access such data Java keystore credential store system is used

Notes:

• CSTORE_ID - Unique ID of the credential store to use. Should be "jks"
• NAME - Credential's name of the secret to add to the Java keystore credential store
• VALUE - Credential's value to add to the Java keystore credential store
• USER_GROUP - Group that a user must belong to in order to access the secret

1. Create a credential inside the streamsets instance

$SDC_DIST/bin/streamsets stagelib-cli jks-credentialstore add -i <CSTORE_ID> -n <NAME> -c <VALUE>

# example credential create
$SDC_DIST/bin/streamsets stagelib-cli jks-credentialstore add -i jks -n mongopass -c password

# example list credentials by CSTORE_ID
$SDC_DIST/bin/streamsets stagelib-cli jks-credentialstore list -i jks

2. Replace credential value by special credential:get() function

${credential:get("<STORE_ID>", "<USER_GROUP>", "<NAME>")}

# example
${credential:get("jks", "all", "mongopass")}

[
{
    "type": "mongo",
    "name": "mongo_source",
    "config": {
      "configBean.mongoConfig.connectionString": "mongodb://mongo:27017",
      "configBean.mongoConfig.database": "db_name",
      "configBean.mongoConfig.collection": "collection_name",
      "configBean.mongoConfig.username": "user",
      "configBean.mongoConfig.password": "${credential:get(\"jks\", \"all\", \"mongopass\")}",
      "configBean.mongoConfig.authSource": "admin"
    }
  }
]