Guide to ephemeral cloudigrade - cloudigrade/cloudigrade GitHub Wiki

This document is a beginner's guide/tutorial/HOWTO for working with cloudigrade deployed to an ephemeral cluster namespace. This document will show you how to work with AWS, sources-api, and cloudigrade to get real AWS data in your cloudigrade deployment.

Prerequisites

This guide assumes you already are able to run cloudigrade, are able to work with bonfire and the ephemeral cluster, and have separate customer/tenant AWS account and Azure subscription. See also: development environment and Ephemeral Cluster Deployment.

Deploy

Reserve an ephemeral namespace if you don't already have one.

export NAMESPACE=$(yes | bonfire namespace reserve --duration 8h --pool real-managed-kafka | grep -o 'ephemeral-[a-zA-Z0-9]*$')
echo "${NAMESPACE}

Set the oc project to the reserved namespace.

oc project "${NAMESPACE}"

Set any necessary environment variables and deploy cloudigrade.

source ~/.env-files/ephemeral
ansible-playbook \
    -e namespace="${NAMESPACE}" \
    -e env="${CLOUDIGRADE_ENVIRONMENT}" \
    deployment/playbooks/manage-clowder.yml

Wait for deployment to complete.

Set env vars for local commands

We will set some environment variables here for convenient reuse. These are not strictly required, but they make later commands more legible.

In a later step, we will port-forward cloudigrade-api to localhost:8000 and sources-api to localhost:8002. Adjust the port numbers here if needed.

LOCAL_CLOUDIGRADE_PORT="8000"
LOCAL_SOURCES_API_PORT="8002"
CLOUDIGRADE_INTERNAL_URL=":${LOCAL_CLOUDIGRADE_PORT}/internal"
CLOUDIGRADE_BASE_URL=":${LOCAL_CLOUDIGRADE_PORT}/api/cloudigrade/v2"
SOURCES_BASE_URL=":${LOCAL_SOURCES_API_PORT}/api/sources/v3.1"

# define the identity of the user that will make requests.
# the actual numerical values don't really matter since sources and cloudigrade will trust the incoming x-rh-identity header.
RH_USER_ACCOUNT_NUMBER="1234"
RH_USER_ORG_ID="5678"
X_RH_IDENTITY=$(echo '{"identity":{"account_number":"'"${RH_USER_ACCOUNT_NUMBER}"'","org_id":"'"${RH_USER_ORG_ID}"'"}}' | base64)

# get the PSK for making requests to sources-api.
# if you look at the value, it should be something like 'thisMustBeEphemeralOrMinikube'.
# that's a special hard-coded secret for ephemeral; stage and prod deployments are long random strings.
SOURCES_PSK=$(oc get secrets/sources-api-secrets -o jsonpath='{.data.psks}' | base64 -d)

# define headers specifically for sources-api HTTP requests.
HEADER_SOURCES_PSK="x-rh-sources-psk:${SOURCES_PSK}"
HEADER_SOURCES_ACCOUNT_NUMBER="x-rh-sources-account-number:${RH_USER_ACCOUNT_NUMBER}"

oc port-forward

These may need to be repeated if openshift kills your connections or pods. That happens a lot.

CLOUDIGRADE_API_POD_NAME=$(oc get pods -o jsonpath='{.items[?(.status.phase=="Running")].metadata.name}' -l pod=cloudigrade-api | awk '{print $1}')
oc port-forward pods/"${CLOUDIGRADE_API_POD_NAME}" "${LOCAL_CLOUDIGRADE_PORT}":8000 2>&1 >/dev/null &

SOURCES_API_SVC_POD_NAME=$(oc get pods -o jsonpath='{.items[?(.status.phase=="Running")].metadata.name}' -l pod=sources-api-svc | awk '{print $1}')
oc port-forward pods/"${SOURCES_API_SVC_POD_NAME}" "${LOCAL_SOURCES_API_PORT}":8000 2>&1 >/dev/null &

Create customer AWS policy and role

Normally these instructions would be shown to the customer in the web UI, or they'd be performed automatically by sources, but the ephemeral deployment does not currently include the UI, and we want to do this manually. If you intend to reuse the same customer AWS account, you probably only need to do this once as long as you save the created ARN somewhere.

  • As the customer, go to https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-east-1#/policies and click "Create policy"
  • Click the JSON tab.
  • Switch back to your terminal, and run the following command. On macOS, pbcopy takes stdin and puts it on your clipboard. Then you can simply paste it like regular text. If you're not using macOS, use whatever is your platform equivalent to copy the output. 
    http "${CLOUDIGRADE_BASE_URL}/sysconfig/" "X-RH-IDENTITY:${X_RH_IDENTITY}" | jq .aws_policies.traditional_inspection | pbcopy
  • Switch back to your web browser, and paste the text to replace the empty policy JSON.
  • Proceed to the "Review policy" page.
  • Give the policy a unique and memorable name, and click "Create policy".
  • Go to https://us-east-1.console.aws.amazon.com/iamv2/home#/roles and click "Create role".
  • Select trusted entity type "AWS account" and then "Another AWS account".
  • Switch back to your terminal, and run the following command. Again, replace pbcopy if necessary. 
    http "${CLOUDIGRADE_BASE_URL}/sysconfig/" "X-RH-IDENTITY:${X_RH_IDENTITY}" | jq -r .aws_account_id | pbcopy
  • Switch back to your web browser, and paste the text into the Account ID text box.
  • Important note: click your username in the top right corner of the page, and compare the Account ID there with the one you just pasted. They should be different! You should be logged in as the customer AWS account, and the value you pasted from the cloudigrade API should be a different account acting as the service provider.
  • Proceed to the "Add permissions" page.
  • Find the policy you created a few minutes ago, and check the box by it.
  • Proceed to the "Name, review, and create" page.
  • Give the role a unique and memorable name, and click "Create role".
  • Once created, click "View role".
  • Click the page icon under "ARN" to copy the ARN to your clipboard.
  • Switch back to your terminal, and set the ARN environment variable to what you copied. 
    # for example
ARN='arn:aws:iam::000000000000:role/my-great-example-role'

Create source for customer AWS account

First we have to get the application type ID because the value may change between deployments. Remove the noglob if you are not using zsh.

APPLICATION_TYPE="/insights/platform/cloud-meter"
APPLICATION_TYPES_RESPONSE=$( \
    noglob http "${SOURCES_BASE_URL}/application_types" \
    'filter[name]'=="${APPLICATION_TYPE}" \
    "${HEADER_SOURCES_PSK}" "${HEADER_SOURCES_ACCOUNT_NUMBER}" \
)
APPLICATION_TYPE_ID=$(echo $APPLICATION_TYPES_RESPONSE | jq -r '.data[0].id')

Next, create the source, application, and authentication objects using one call to the "bulk create" API.

SOURCE_NAME="cloudmeter-source-$RANDOM"
BULK_CREATE_MESSAGE='{
   "sources":[
      {
         "name":"'"${SOURCE_NAME}"'",
         "app_creation_workflow":"manual_configuration",
         "source_type_name":"amazon"
      }
   ],
   "authentications":[
      {
         "authtype":"cloud-meter-arn",
         "username":"'"${ARN}"'",
         "resource_type":"application",
         "resource_name":"/insights/platform/cloud-meter"
      }
   ],
   "applications":[
      {
         "application_type_id":"'"${APPLICATION_TYPE_ID}"'",
         "source_name":"'"${SOURCE_NAME}"'"
      }
   ]
}'
BULK_CREATE_RESPONSE=$( \
    echo "$BULK_CREATE_MESSAGE" | \
    http post "${SOURCES_BASE_URL}/bulk_create" \
    "${HEADER_SOURCES_PSK}" "${HEADER_SOURCES_ACCOUNT_NUMBER}" \
)

Extract these IDs from the response so we can interrogate the individual created objects.

APPLICATION_ID=$(echo $BULK_CREATE_RESPONSE | jq -r ".applications[0].id")
SOURCE_ID=$(echo $BULK_CREATE_RESPONSE | jq -r ".sources[0].id")

Repeat the following commands either until you see the status become "available" (happy path) or a few minutes have passed (maybe something went wrong).

# get the application status
http "${SOURCES_BASE_URL}/applications/${APPLICATION_ID}" \
    "${HEADER_SOURCES_PSK}" "${HEADER_SOURCES_ACCOUNT_NUMBER}"

# get the source status
http "${SOURCES_BASE_URL}/sources/${SOURCE_ID}" \
    "${HEADER_SOURCES_PSK}" "${HEADER_SOURCES_ACCOUNT_NUMBER}"

Check the cloudigrade API

When the source and application are available, that means cloudigrade successfully created, verified, and enabled the cloud account.

Check various cloudigrade public APIs as the customer/user identity.

# check the cloudigrade cloud account.
# you should see one result with your ARN.
http "${CLOUDIGRADE_BASE_URL}"/accounts/ \
    "X-RH-IDENTITY:${X_RH_IDENTITY}"

# check the cloudigrade instances.
# if you were running any instance in your customer AWS account, you should see them here.
http "${CLOUDIGRADE_BASE_URL}"/instances/ \
    "X-RH-IDENTITY:${X_RH_IDENTITY}"

# check the cloudigrade machine images.
# if you were running any instance in your customer AWS account, you should see their images here.
http "${CLOUDIGRADE_BASE_URL}"/images/ \
    "X-RH-IDENTITY:${X_RH_IDENTITY}"

Since you're port-forwarded into the pod, you can also request the internal API without customer/user authentication to get all data (though it's probably still just your one user at this point). Some you may want to try:

http "${CLOUDIGRADE_INTERNAL_URL}/api/cloudigrade/v1/users/"
http "${CLOUDIGRADE_INTERNAL_URL}/api/cloudigrade/v1/machineimages/"
http "${CLOUDIGRADE_INTERNAL_URL}/api/cloudigrade/v1/awsmachineimages/"
http "${CLOUDIGRADE_INTERNAL_URL}/api/cloudigrade/v1/azuremachineimages/"
http "${CLOUDIGRADE_INTERNAL_URL}/api/cloudigrade/v1/cloudaccounts/"
http "${CLOUDIGRADE_INTERNAL_URL}/api/cloudigrade/v1/awscloudaccounts/"
http "${CLOUDIGRADE_INTERNAL_URL}/api/cloudigrade/v1/azurecloudaccounts/"
http "${CLOUDIGRADE_INTERNAL_URL}/api/cloudigrade/v1/instances/"
http "${CLOUDIGRADE_INTERNAL_URL}/api/cloudigrade/v1/awsinstances/"
http "${CLOUDIGRADE_INTERNAL_URL}/api/cloudigrade/v1/azureinstances/"

Direct database access

If you have a local postgres client and want to connect directly to the database for cloudigrade in the ephemeral deployment, you should port-forward to the postigrade pod and get several generated secrets.

# get generated secrets from openshift
POSTIGRADE_SVC_POD_NAME=$(oc get pods -o jsonpath='{.items[?(.status.phase=="Running")].metadata.name}' -l pod=postigrade-svc  | awk '{print $1}') && echo "POSTIGRADE_SVC_POD_NAME ${POSTIGRADE_SVC_POD_NAME}"
oc get secrets/cloudigrade-app-secret -o jsonpath='{.data.cloudigrade-psks}' | base64 -d > /tmp/psks.json
oc get secrets/postigrade -o jsonpath='{.data.cdappconfig\.json}' | base64 -d > /tmp/cdappconfig.json

# export secrets to local env
export PGDATABASE=$(jq -r .database.name /tmp/cdappconfig.json)
export PGUSER=$(jq -r .database.username /tmp/cdappconfig.json)
export PGPASSWORD=$(jq -r .database.password /tmp/cdappconfig.json)
export PGPORT=8001
export PGHOST=127.0.0.1

# port-forward psql
oc port-forward pods/"${POSTIGRADE_SVC_POD_NAME}" 8001:8000 2>&1 > /dev/null &

# psql will automatically use the env vars exported above
psql

Cleanup

It's important that you explicitly delete the source instead of just letting bonfire and clowder destroy the deployment. If you don't destroy the source, then the customer AWS account's CloudTrail will continue feeding data to you service provider AWS account's S3 bucket, and this may put extra load on your future cloudigrade deployments.

http delete "${SOURCES_BASE_URL}/sources/${SOURCE_ID}" "${HEADER_SOURCES_PSK}" "${HEADER_SOURCES_ACCOUNT_NUMBER}"

If you forget or are unable to delete the source, as the customer AWS account: