Sossity Terraform Output

Sossity's main purpose is to create a .tf.json (Terraform) file of all the necessary cloud resources, and, essentially, which resources depend on each other. This guarantees that a cluster will always have end-to-end paths when built, and not lose messages. This file is then read by terraform apply on the CircleCI node, which uses it to build the resources.

Many of these resources are identically named to the config.clj items. This document only highlights when the TF representation is significantly different. See Sossity Config File.

Note: this needs to be well-formatted JSON, so everything is enclosed in {}

NEVER EDIT THIS FILE

Terraform sections

Full demo file at end.

Universal/Standard:

"depends_on":[] -- an array of terraform resource names which this resource depends on. Terraform builds these in the order of dependency, so there are no 'orphan' resources.

provider

{
  "provider" : {
    "google" : { "credentials":"","project":"","region":""},
    "googlecli": {},
    "googleappengine": {},
    "googlebigquery" : {}}}

Google Cloud credentials and configuration for different resource types. google uses the standard Google Terraform module. googlecli uses a 22acacia built module for the gcloud CLI command. googleappengine uses the App Engine Go library. googlebigquery uses the BigQuery Go library.

resources : google_pubsub_topic

Represents a Pubsub Topic.

 "google_pubsub_topic" : {
      "topic-name" : {
        "name" : ""}}

google_pubsub_topic - Terraform resource type

topic-name:{} - internal Pubsub Topic name, same as ...

{"name" :"topic-name"} - Pubsub Topic name on Google

resources: google_pubsub_subscription

Represents a single subscription to a Pubsub Topic. Currently used only by Sinks. (Dataflow jobs create their own subscriptions internally).

    "google_pubsub_subscription" : {
      "sub-name" : {
        "name" : "",
        "topic" : ""}}

sub-name - internal name of subscription, always the topic postfixed with _sub

name: Name of the subscription in Google. Same as sub-name.

topic: Name of the topic in Google.

resources: google_container_cluster

Represents a Kubernetes cluster. See :cluster section in the Sossity Config File for more information.

resources: googlecli_container_replica_controller

Represents a Docker container in the cluster.

    "googlecli_container_replica_controller" : {
      "sink-name" : {
        "name" : "sink-name",
        "resource_version" : [ "1" ],
        "docker_image" : "",
        "container_name" : "",
        "zone" : "",
        "env_args" : {
          "num_retries" : 3,
          "batch_size" : 1000,
          "proj_name" : "",
          "sub_name" : "",
          "bucket_name" : ""}}

sink-name : internal resource name. same as Cloud resource name.

name: same as sink-name

resource-version - hardcoded in config.clj. Increment to force redeploy all Sinks.

docker-image - Image in Google Docker repository

container_name - same as cluster name

zone - Instance region

env_args:{} - Environment Variables passed to Docker container, read by Sink program. Can be anything.

num_retries - how many times to re-try writing a row to Cloud Storage before failure

batch_size - number of lines / file flushed to Cloud Storage

proj_name - Google Cloud project

sub_name - Pubsub Topic subscription

bucket_name - Cloud storage output bucket

resources: "googleappengine_app"

Represents a Source: App Engine REST endpoint

    "googleappengine_app" : {
      "module-name" : {
        "moduleName" : "",
        "version" : "init",
        "gstorageKey" : "",
        "resource_version" : [ "1" ],
        "gstorageBucket" : "",
        "scaling" : {
          "minIdleInstances" : 1,
          "maxIdleInstances" : 1,
          "minPendingLatency" : "",
          "maxPendingLatency" : ""
        },
        "topicName" : ""}

module-name - Internal module name, same as moduleName

modeuleName - App Engine module name

version - always 'init' so new deploys roll out over old ones

gstorageKey - key for where Source Jar is stored

resource_version - similar to Sink, increment to force new Source deploy

scaling - various App Engine scaling settings, see App Engine documentation

topicName - output Pubsub topic

resource: google_storage_bucket

Represents a Google Cloud Storage bucket

    "google_storage_bucket" : {
      "bucket-name" : {
        "name" : "",
        "force_destroy" : true,
        "location" : ""}

bucket-name: internal bucket name, same as name

name: Google Cloud Storage bucket name

force_destroy: if true, do not delete if terraform resource deleted

location: bucket region: usually EU, US-WEST, etc

resource: googlecli_dataflow

Represents a Google Cloud Dataflow resource. Exact translation of config variables, see Sossity Config File.

resource: googlebigquery_dataset

Represents a Big Query dataset resource. Exact translation of config variables, see Sossity Config File.

resource: googlebigquery_table

Represents a Big Query table resource. Exact translation of config variables, see Sossity Config File.

Full Example

{
  "provider" : {
    "google" : {
      "credentials" : "${file(\"/home/ubuntu/demo-config/account.json\")}",
      "project" : "hx-test",
      "region" : "europe-west1-c"
    },
    "googlecli" : {
      "credentials" : "${file(\"/home/ubuntu/demo-config/account.json\")}",
      "project" : "hx-test",
      "region" : "europe-west1-c"
    },
    "googleappengine" : {
      "credentials" : "${file(\"/home/ubuntu/demo-config/account.json\")}",
      "project" : "hx-test",
      "region" : "europe-west1-c"
    },
    "googlebigquery" : {
      "credentials" : "${file(\"/home/ubuntu/demo-config/account.json\")}",
      "project" : "hx-test",
      "region" : "europe-west1-c"
    }
  },
  "resource" : {
    "google_pubsub_topic" : {
      "testendpoint-to-testpipeline" : {
        "name" : "testendpoint-to-testpipeline"
      },
      "orionbq-to-orionbq-error" : {
        "name" : "orionbq-to-orionbq-error"
      },
      "testpipeline-to-testsink" : {
        "name" : "testpipeline-to-testsink"
      },
      "orionsink-to-orionsink-error" : {
        "name" : "orionsink-to-orionsink-error"
      },
      "testsink-to-testsink-error" : {
        "name" : "testsink-to-testsink-error"
      },
      "orionidentitypipe-to-orionsink" : {
        "name" : "orionidentitypipe-to-orionsink"
      },
      "orionbqfilter-to-orionbq" : {
        "name" : "orionbqfilter-to-orionbq"
      },
      "orion-to-orionidentitypipe" : {
        "name" : "orion-to-orionidentitypipe"
      },
      "orionbqfilter-to-orionbqfilter-error" : {
        "name" : "orionbqfilter-to-orionbqfilter-error"
      },
      "testpipeline-to-testpipeline-error" : {
        "name" : "testpipeline-to-testpipeline-error"
      },
      "orion-to-orionbqfilter" : {
        "name" : "orion-to-orionbqfilter"
      },
      "orionidentitypipe-to-orionidentitypipe-error" : {
        "name" : "orionidentitypipe-to-orionidentitypipe-error"
      }
    },
    "google_pubsub_subscription" : {
      "orionidentitypipe-to-orionsink_sub" : {
        "name" : "orionidentitypipe-to-orionsink_sub",
        "topic" : "orionidentitypipe-to-orionsink",
        "depends_on" : [ "google_pubsub_topic.orionidentitypipe-to-orionsink" ]
      },
      "orionidentitypipe-to-orionidentitypipe-error_sub" : {
        "name" : "orionidentitypipe-to-orionidentitypipe-error_sub",
        "topic" : "orionidentitypipe-to-orionidentitypipe-error",
        "depends_on" : [ "google_pubsub_topic.orionidentitypipe-to-orionidentitypipe-error" ]
      },
      "testsink-to-testsink-error_sub" : {
        "name" : "testsink-to-testsink-error_sub",
        "topic" : "testsink-to-testsink-error",
        "depends_on" : [ "google_pubsub_topic.testsink-to-testsink-error" ]
      },
      "orionbq-to-orionbq-error_sub" : {
        "name" : "orionbq-to-orionbq-error_sub",
        "topic" : "orionbq-to-orionbq-error",
        "depends_on" : [ "google_pubsub_topic.orionbq-to-orionbq-error" ]
      },
      "testpipeline-to-testpipeline-error_sub" : {
        "name" : "testpipeline-to-testpipeline-error_sub",
        "topic" : "testpipeline-to-testpipeline-error",
        "depends_on" : [ "google_pubsub_topic.testpipeline-to-testpipeline-error" ]
      },
      "orionbqfilter-to-orionbqfilter-error_sub" : {
        "name" : "orionbqfilter-to-orionbqfilter-error_sub",
        "topic" : "orionbqfilter-to-orionbqfilter-error",
        "depends_on" : [ "google_pubsub_topic.orionbqfilter-to-orionbqfilter-error" ]
      },
      "testpipeline-to-testsink_sub" : {
        "name" : "testpipeline-to-testsink_sub",
        "topic" : "testpipeline-to-testsink",
        "depends_on" : [ "google_pubsub_topic.testpipeline-to-testsink" ]
      },
      "orionsink-to-orionsink-error_sub" : {
        "name" : "orionsink-to-orionsink-error_sub",
        "topic" : "orionsink-to-orionsink-error",
        "depends_on" : [ "google_pubsub_topic.orionsink-to-orionsink-error" ]
      }
    },
    "google_container_cluster" : {
      "hx_fstack_cluster" : {
        "name" : "hxhstack",
        "initial_node_count" : 4,
        "master_auth" : {
          "username" : "hx",
          "password" : "hstack"
        },
        "node_config" : {
          "oauth_scopes" : [ "https://www.googleapis.com/auth/compute", "https://www.googleapis.com/auth/devstorage.read_only", "https://www.googleapis.com/auth/logging.write", "https://www.googleapis.com/auth/monitoring", "https://www.googleapis.com/auth/cloud-platform" ],
          "machine_type" : "n1-standard-1"
        },
        "zone" : "europe-west1-c"
      }
    },
    "googlecli_container_replica_controller" : {
      "orionsink-sink" : {
        "name" : "orionsink-sink",
        "resource_version" : [ "1" ],
        "docker_image" : "gcr.io/hx-test/store-sink",
        "container_name" : "${google_container_cluster.hx_fstack_cluster.name}",
        "zone" : "europe-west1-c",
        "env_args" : {
          "num_retries" : 3,
          "batch_size" : 1000,
          "proj_name" : "hx-test",
          "sub_name" : "orionidentitypipe-to-orionsink_sub",
          "bucket_name" : "orionsinkbucket"
        }
      },
      "testsink-sink" : {
        "name" : "testsink-sink",
        "resource_version" : [ "1" ],
        "docker_image" : "gcr.io/hx-test/store-sink",
        "container_name" : "${google_container_cluster.hx_fstack_cluster.name}",
        "zone" : "europe-west1-c",
        "env_args" : {
          "num_retries" : 3,
          "batch_size" : 1000,
          "proj_name" : "hx-test",
          "sub_name" : "testpipeline-to-testsink_sub",
          "bucket_name" : "testsink-bucket"
        }
      }
    },
    "googleappengine_app" : {
      "testendpoint" : {
        "moduleName" : "testendpoint",
        "version" : "init",
        "gstorageKey" : "hxtest-1.0-SNAPSHOT",
        "resource_version" : [ "1" ],
        "gstorageBucket" : "build-artifacts-public-eu",
        "scaling" : {
          "minIdleInstances" : 1,
          "maxIdleInstances" : 1,
          "minPendingLatency" : "3s",
          "maxPendingLatency" : "6s"
        },
        "topicName" : "projects/hx-test/topics/testendpoint-to-testpipeline"
      },
      "orion" : {
        "moduleName" : "orion",
        "version" : "init",
        "gstorageKey" : "hxtest-1.0-SNAPSHOT",
        "resource_version" : [ "1" ],
        "gstorageBucket" : "build-artifacts-public-eu",
        "scaling" : {
          "minIdleInstances" : 1,
          "maxIdleInstances" : 1,
          "minPendingLatency" : "3s",
          "maxPendingLatency" : "6s"
        },
        "topicName" : "projects/hx-test/topics/orion-to-orionbqfilter"
      }
    },
    "google_storage_bucket" : {
      "orionsinkbucket" : {
        "name" : "orionsinkbucket",
        "force_destroy" : true,
        "location" : "EU"
      },
      "orionidentitypipe-error" : {
        "name" : "orionidentitypipe-error",
        "force_destroy" : true,
        "location" : "EU"
      },
      "testsink-error" : {
        "name" : "testsink-error",
        "force_destroy" : true,
        "location" : "EU"
      },
      "orionbq-error" : {
        "name" : "orionbq-error",
        "force_destroy" : true,
        "location" : "EU"
      },
      "testpipeline-error" : {
        "name" : "testpipeline-error",
        "force_destroy" : true,
        "location" : "EU"
      },
      "orionbqfilter-error" : {
        "name" : "orionbqfilter-error",
        "force_destroy" : true,
        "location" : "EU"
      },
      "testsink-bucket" : {
        "name" : "testsink-bucket",
        "force_destroy" : true,
        "location" : "EU"
      },
      "orionsink-error" : {
        "name" : "orionsink-error",
        "force_destroy" : true,
        "location" : "EU"
      }
    },
    "googlecli_dataflow" : {
      "orionbq" : {
        "name" : "orionbq",
        "classpath" : "/usr/local/lib/angleddream-bundled-0.1-ALPHA.jar",
        "class" : "com.acacia.angleddream.Main",
        "depends_on" : [ "googlecli_dataflow.orionbqfilter", "google_pubsub_topic.orionbq-to-orionbq-error", "google_pubsub_topic.orionbqfilter-to-orionbq" ],
        "optional_args" : {
          "stagingLocation" : "gs://hx-test/staging-eu",
          "zone" : "europe-west1-c",
          "workerMachineType" : "n1-standard-1",
          "bigQueryTable" : "hx_orion",
          "errorPipelineName" : "projects/hx-test/topics/orionbq-to-orionbq-error",
          "bigQueryDataset" : "hx_orion_staging",
          "pubsubTopic" : "projects/hx-test/topics/orionbqfilter-to-orionbq",
          "numWorkers" : "1",
          "bigQuerySchema" : "/home/ubuntu/demo-config/orion.json",
          "pipelineName" : "orionbq",
          "maxNumWorkers" : "1"
        }
      },
      "orionbqfilter" : {
        "name" : "orionbqfilter",
        "classpath" : "/usr/local/lib/angleddream-bundled-0.1-ALPHA.jar:/usr/local/lib/orion-transform-0.1-ALPHA.jar",
        "class" : "com.acacia.angleddream.Main",
        "depends_on" : [ "google_pubsub_topic.orionbqfilter-to-orionbq", "googleappengine_app.orion", "google_pubsub_topic.orionbqfilter-to-orionbqfilter-error", "google_pubsub_topic.orion-to-orionbqfilter" ],
        "optional_args" : {
          "pubsubTopic" : "projects/hx-test/topics/orion-to-orionbqfilter",
          "pipelineName" : "orionbqfilter",
          "errorPipelineName" : "projects/hx-test/topics/orionbqfilter-to-orionbqfilter-error",
          "outputTopics" : "projects/hx-test/topics/orionbqfilter-to-orionbq",
          "maxNumWorkers" : "1",
          "numWorkers" : "1",
          "zone" : "europe-west1-c",
          "workerMachineType" : "n1-standard-1",
          "stagingLocation" : "gs://hx-test/staging-eu"
        }
      },
      "orionidentitypipe" : {
        "name" : "orionidentitypipe",
        "classpath" : "/usr/local/lib/angleddream-bundled-0.1-ALPHA.jar:/usr/local/lib/identitypipeline-0.1-ALPHA.jar",
        "class" : "com.acacia.angleddream.Main",
        "depends_on" : [ "google_pubsub_topic.orionidentitypipe-to-orionsink", "googleappengine_app.orion", "google_pubsub_topic.orionidentitypipe-to-orionidentitypipe-error", "google_pubsub_topic.orion-to-orionidentitypipe" ],
        "optional_args" : {
          "pubsubTopic" : "projects/hx-test/topics/orion-to-orionidentitypipe",
          "pipelineName" : "orionidentitypipe",
          "errorPipelineName" : "projects/hx-test/topics/orionidentitypipe-to-orionidentitypipe-error",
          "outputTopics" : "projects/hx-test/topics/orionidentitypipe-to-orionsink",
          "maxNumWorkers" : "1",
          "numWorkers" : "1",
          "zone" : "europe-west1-c",
          "workerMachineType" : "n1-standard-1",
          "stagingLocation" : "gs://hx-test/staging-eu"
        }
      },
      "testpipeline" : {
        "name" : "testpipeline",
        "classpath" : "/usr/local/lib/angleddream-bundled-0.1-ALPHA.jar:/usr/local/lib/identitypipeline-0.1-ALPHA.jar",
        "class" : "com.acacia.angleddream.Main",
        "depends_on" : [ "google_pubsub_topic.testpipeline-to-testsink", "googleappengine_app.testendpoint", "google_pubsub_topic.testpipeline-to-testpipeline-error", "google_pubsub_topic.testendpoint-to-testpipeline" ],
        "optional_args" : {
          "pubsubTopic" : "projects/hx-test/topics/testendpoint-to-testpipeline",
          "pipelineName" : "testpipeline",
          "errorPipelineName" : "projects/hx-test/topics/testpipeline-to-testpipeline-error",
          "outputTopics" : "projects/hx-test/topics/testpipeline-to-testsink",
          "maxNumWorkers" : "1",
          "numWorkers" : "1",
          "zone" : "europe-west1-c",
          "workerMachineType" : "n1-standard-1",
          "stagingLocation" : "gs://hx-test/staging-eu"
        }
      }
    },
    "googlebigquery_dataset" : {
      "hx_orion_staging" : {
        "datasetId" : "hx_orion_staging"
      }
    },
    "googlebigquery_table" : {
      "hx_orion" : {
        "tableId" : "hx_orion",
        "depends_on" : [ "googlebigquery_dataset.hx_orion_staging" ],
        "datasetId" : "${googlebigquery_dataset.hx_orion_staging.datasetId}",
        "schemaFile" : "/home/ubuntu/demo-config/orion.json"
      }
    }
  }}