Sources Integration - cloudigrade/cloudigrade GitHub Wiki

How do I create "sources"?

cloudigrade v2 relies on the Insights platform to have created sources to feed into cloudigrade's cloud account creation and deletion. To create a source in the Insights UI, try the following pages:

Alternatively, you may interact directly with sources-api if you have also access to it. For a demonstration of this interaction (as of 2020-03-26), see https://asciinema.org/a/313796 (starting around 00:14).

Sources API calls

Variables setup for API calls

These aliases and environment variables are not strictly required; they simply make the http commands more concise and DRY.

read -s AUTH  # HTTP basic credentials in the form of username:password
read -s ARN  # the ARN for the AWS source you're going to set up

alias HTTP_PROD="http --auth ${AUTH}"
alias HTTP_STAGE="http --auth ${AUTH} --proxy https:http://squid.corp.redhat.com:3128"
alias HTTP_CI_QA="http --verify=no --auth ${AUTH}"

PROD_BASE_URL="https://cloud.redhat.com/api"
STAGE_BASE_URL="https://cloud.stage.redhat.com/api"
CI_BASE_URL="https://ci.cloud.redhat.com/api"
QA_BASE_URL="https://qa.cloud.redhat.com/api"

# change these two according to your environment
alias HTTP=HTTP_STAGE  
BASE_URL=$STAGE_BASE_URL

SOURCES_BASE_URL="${BASE_URL}/sources/v3.1"
SOURCE_NAME="cloudmeter-demo"  # set to whatever you like, but it must be unique within the scope of your RH login

Sources separate calls to create a cloud account

These commands each tee their http output to a file under /tmp so you can reference them in case of unexpected behaviors. If you are using these commands many times, consider using tee -a or something else to keep track of the responses.

Important note: The APPLICATIONTYPEID and SOURCETYPEID are not statically defined type values. They may vary between deployments and environments, and you should always get their current values from the API.

APPLICATIONTYPEID=$(HTTP "${SOURCES_BASE_URL}/application_types" \
'filter[name]'==/insights/platform/cloud-meter | \
tee /tmp/APPLICATIONTYPE.json | jq -r '.data[0].id')
echo "APPLICATIONTYPEID=${APPLICATIONTYPEID}"

SOURCETYPEID=$(HTTP "${SOURCES_BASE_URL}/source_types" \
'filter[name]'=="amazon" | \
tee /tmp/SOURCETYPE.json | jq -r '.data[0].id')
echo "SOURCETYPEID=${SOURCETYPEID}"

SOURCEID=$(HTTP "${SOURCES_BASE_URL}/sources" \
name="${SOURCE_NAME}" source_type_id="${SOURCETYPEID}" | \
tee /tmp/SOURCE.json | jq -r '.id')
echo "SOURCEID=${SOURCEID}"

APPLICATIONID=$(HTTP "${SOURCES_BASE_URL}/applications" \
application_type_id="${APPLICATIONTYPEID}" source_id="${SOURCEID}" | \
tee /tmp/APPLICATION.json | jq -r '.id')
echo "APPLICATIONID=${APPLICATIONID}"

AUTHENTICATIONID=$(HTTP "${SOURCES_BASE_URL}/authentications" \
resource_type=Application resource_id="${APPLICATIONID}" password="${ARN}" authtype=cloud-meter-arn | \
tee /tmp/AUTHENTICATION.json | jq -r '.id')
echo "AUTHENTICATIONID=${AUTHENTICATIONID}"

APPLICATIONAUTHENTICATIONID=$(HTTP "${SOURCES_BASE_URL}/application_authentications" \
application_id="${APPLICATIONID}" authentication_id="${AUTHENTICATIONID}" | \
tee /tmp/APPLICATIONAUTHENTICATION.json | jq -r '.id')
echo "APPLICATIONAUTHENTICATIONID=${APPLICATIONAUTHENTICATIONID}"

Sources "bulk create" alternative

As an alternative to the numerous API calls above, you can use a bulk API to reduce the number of individual calls to define these objects.

APPLICATIONTYPEID=$(HTTP "${SOURCES_BASE_URL}/application_types" \
'filter[name]'==/insights/platform/cloud-meter | \
tee /tmp/APPLICATIONTYPE.json | jq -r '.data[0].id')
echo "APPLICATIONTYPEID=${APPLICATIONTYPEID}"

echo '{
   "sources":[
      {
         "name":"'"${SOURCE_NAME}"'",
         "app_creation_workflow":"manual_configuration",
         "source_type_name":"amazon"
      }
   ],
   "authentications":[
      {
         "authtype":"cloud-meter-arn",
         "username":"'"${ARN}"'",
         "resource_type":"application",
         "resource_name":"/insights/platform/cloud-meter"
      }
   ],
   "applications":[
      {
         "application_type_id":"'"${APPLICATIONTYPEID}"'",
         "source_name":"'"${SOURCE_NAME}"'"
      }
   ]
}' | jq > /tmp/BULKCREATE_postdata.json
cat /tmp/BULKCREATE_postdata.json | HTTP POST "${SOURCES_BASE_URL}/bulk_create" | tee /tmp/BULKCREATE.json

APPLICATIONID=$(cat /tmp/BULKCREATE.json | jq -r ".applications[0].id")
echo "APPLICATIONID=${APPLICATIONID}"
SOURCEID=$(cat /tmp/BULKCREATE.json | jq -r ".sources[0].id")
echo "SOURCEID=${SOURCEID}"

Check the application status

After creation, cloudigrade may update the application object's status to indicate setup success or failure.

HTTP "${SOURCES_BASE_URL}/applications/${APPLICATIONID}"

Sources calls to update a cloud account

HTTP patch "${SOURCES_BASE_URL}/authentications/${AUTHENTICATIONID}" password="${ARN}" | \
tee /tmp/AUTHENTICATION_updated.json

Sources calls to delete a cloud account

This one delete should cascade to delete the related objects you created:

HTTP delete "${SOURCES_BASE_URL}/sources/${SOURCEID}"

In case you want to verify that they were actually deleted:

HTTP "${SOURCES_BASE_URL}/application_authentications/${APPLICATIONAUTHENTICATIONID}"
HTTP "${SOURCES_BASE_URL}/authentications/${AUTHENTICATIONID}"
HTTP "${SOURCES_BASE_URL}/applications/${APPLICATIONID}"
HTTP "${SOURCES_BASE_URL}/sources/${SOURCEID}"

Kafka topics from sources-api

Each of the above API calls to sources will cause sources to post messages to a Kafka topic cloudigrade is subscribed to. cloudigrade reads from the following Kafka topics and relies on sources-api to put appropriate messages on them:

  • ApplicationAuthentication.create
  • Authentication.update
  • ApplicationAuthentication.destroy

clouigrade will only attempt to create a CloudAccount based on messages from ApplicationAuthentication.create. cloudigrade must verify that appropriate objects exist in sources-api, and if any of the following conditions is true, the CloudAccount will not be created and the message is effectively ignored:

  • Application is not found for application_id for account_number.
  • Application type is not "cloudmeter".
  • Authentication is not found for authentication_id and account number.
  • Authentication authtype is not "cloud-meter-arn".
  • Authentication resource_type is not "Application"
  • Authentication password is not set.

Following these sources-api checks, the same verification as with the HTTP API is performed in the user's public cloud account. If that process fails, cloudigrade will not create the CloudAccount and will notify sources-api to update the Application status to "unavailable" with a message describing the cause of failure.

When cloudigrade receives a message from Authentication.update for an AWS account, if the message simply updates a new ARN for the same AWS account, cloudigrade will attempt to verify the ARN permissions with AWS and notify sources-api to update the Application status with either "available" or "unavailable" based on the results. If however the message indicates an ARN for a different AWS account ID, the original CloudAccount will be delete (include all of its Instances, Runs, etc), and a new CloudAccount will be created. In that latter case, all data tracked from the first CloudAccount is lost.

Messages from the ApplicationAuthentication.destroy topics all result in the related CloudAccount being deleted, and all data tracked from the CloudAccount is lost.

Availability Checks

Sources-monitor will kick off an availability_check on unavailable sources that occurs every ~30 minutes.

This availability_check will send a POST request to the /availability_check cloudmeter endpoint. We tell Sources what this endpoint is with an environment variable CLOUD_METER_AVAILABILITY_CHECK_URL in sources (configured here).

The expected payload of this check will be {"source_id": "1234"}

When we receive this call, we will identify the clount with the matching source_id and verify the arn.

Two possible things might happen:

  1. Unavailable Source is still unavailable. We do nothing
  2. Unavailable Source is now available.
    • Mark Clount as enabled.
    • update Authentication and Application (if applicable) availability_status in Sources.
    • If we have not configured cloudtrail for user, do so.
    • If we have not ran initial describe instances, do so.

Cloudigrade also checks to see if available clounts are still available. See periodic permissions checking