This page is for documenting technical solutions or tips to technical issues related to cloud.gov, databases, CircleCI, etc.

• CircleCi pipeline faliures
• Restaging application updates
• Service account passwords
• Hasura password access
• Database backups

It can be very helpful and may be necessary to ssh to the circleci instance where a build pipeline failure occurs. Follow the guidance provided by circleci here and GitHub here to learn how to debug pipeline issues over ssh.

Cloud.gov frequently updates the programming language buildpacks available to customers. Buildpack updates include programming language updates and often include security fixes. Cloud.gov will send an email notification when applications need to be restaged with the subject line of "Action required: restage your application".

A rolling restage operation is the quickest way to upgrade without incurring downtime. You may still want to leverage your deployment infrastructure to perform the upgrade if you have compliance requirements for redeployment operations.

You can restage your applications by opening the command line and entering the following commands:

cf target -o doi-onrr -s dev ; cf restage --strategy rolling dev-nrrd

cf target -o doi-onrr -s dev ; cf restage --strategy rolling dev-blog-nrrd

cf target -o doi-onrr -s dev ; cf restage --strategy rolling preview-data

cf target -o doi-onrr -s dev ; cf restage --strategy rolling dev-onrr-frontend

cf target -o doi-onrr -s dev ; cf restage --strategy rolling dev-onrr-cms

cf target -o doi-onrr -s dev ; cf restage --strategy rolling preview-onrr-frontend

The service account service creates unique cloud.gov credentials for each service key. Service account passwords expire every 90 days. If you see an error like:

Error Code: 403 Raw Response: {"error":"access_denied","error_description":"Your current password has expired. Please reset your password."}

Then you’ll need to delete the existing service key, recreate it, and update the username/password in your deployment scripts. For example:

cf delete-service-key my-service-account my-service-key cf create-service-key my-service-account my-service-key cf service-key my-service-account my-service-key

The last command will return the service account username/password pair. These steps can be used at any time to update/rotate credentials for service accounts.

The username and password for Hasura instances are stored in environment variables vi cf cli. To retrieve the Hasura credentials for the hasura-sandbox app, for example, take the following steps:

1. Target the dev space: cf target -s dev
2. Display environment variables: cf env hasura-sandbox
3. See HASURA_GRAPHQL_ADMIN_SECRET under the User-provided section of the output.

circleci build pipelines currently use a database backup in the Github repository. When database objects are added or modified, a new backup must be taken and committed to the repository in database/backup. Follow the steps outlined below to take a new backup after database changes are applied.

Open a tunnel to the database service with the following command:

cf connect-to-service --no-client <app-name> <rds-service-name>

You will see output similar to this:

Skipping call to client CLI. Connection information:

Host: localhost
Port: <port>
Username: <username>
Password: <password>
Name: <database-name>

Leave this terminal open while you want to use the SSH tunnel. Press Control-C to stop.

In a separate terminal window, run the following command from the database/backup directory of the repository:

pg_dump --user=<username> -W --host=localhost --port=<port> --dbname=<database-name> -Fc -f database_backup.pg

You will be prompted for the user password listed when the tunnel was opened. Paste it in and press enter. The database will be dumped into the database_backup.pg file.

Commit the updated backup file to the repository and push to the remote. Once merged to dev, all subsequent database changes will include the database change you applied.