Integration testing - snowplow-archive/sauna GitHub Wiki

HOME > GUIDE FOR DEVELOPERS > INTEGRATION TESTING

Sauna's integration tests depend on several settings that need to be provided by the developer. Some of them are required and some of them have Snowplow defaults, but can be overridden using environment variables. Here are these settings:

• AWS_ACCESS_KEY_ID - required
• AWS_SECRET_ACCESS_KEY - required
• AWS_BUCKET_NAME - defaults to "sauna-integration-test-bucket-staging"
• AWS_QUEUE_NAME - defaults to "sauna-integration-test-queue-staging"
• AWS_REGION - defaults to "us-east-1"

• OPTIMIZELY_TOKEN - required
• OPTIMIZELY_IMPORT_REGION - defaults to "us-east-1"
• OPTIMIZELY_PROJECT_ID - defaults to "4034532101"
• OPTIMIZELY_SERVICE_ID - defaults to "4034482827"
• OPTIMIZELY_DATASOURCE_ID - defaults to "7082200340"

• SENDGRID_API_KEY_ID - required

Amazon Web Services have first-class support in Sauna. More specifically, we support AWS S3 observer from initial release.

To run integration tests dependent on AWS S3 you may need to set up an IAM Policy that's as restrictive as possible. In order to do this you will need to prepare a S3 bucket which pushes notifications to a configured AWS SQS queue.

First of all, create the bucket and queue:

$ aws s3api create-bucket --bucket sauna-integration-test-bucket

$ aws sqs create-queue --queue-name sauna-integration-test-queue

Please note that AWS resource names are global, so you will need to replace sauna-integration-test-bucket and sauna-integration-test-queue with your actual bucket and queue names.

The next step is to allow S3 to push notifications into the SQS queue using a policy. Create a file called snowplow-sauna-test-policy.json with the following content (012345678912 should be replaced with your 12-digit AWS Account Id, and S3 bucket and SQS queue names should be changed to yours'):

{
    "Policy": "{\"Version\":\"2008-10-17\",\"Id\":\"snowplow-sauna-test-policy\",\"Statement\":[{\"Sid\":\"snowplowanalytics-sauna-queue-sid\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"SQS:SendMessage\",\"Resource\":\"arn:aws:sqs:us-east-1:012345678912:sauna-integration-test-queue\",\"Condition\":{\"ArnLike\":{\"aws:SourceArn\":\"arn:aws:s3:*:*:sauna-integration-test-bucket\"}}}]}"
}

To apply this policy, run the following command:

$ aws sqs set-queue-attributes --queue-url https://queue.amazonaws.com/012345678912/sauna-integration-test-queue --attributes file://snowplow-sauna-test-policy.json

The last step is to make S3 push notifications to the queue. Create another file called snowplow-sauna-test-notification.json with the following content:

{
    "QueueConfigurations": [
        {
            "Id": "snowplow-sauna-test-notification",
            "QueueArn": "arn:aws:sqs:us-east-1:719197435995:sauna-integration-test-queue",
            "Events": [
                "s3:ObjectCreated:*"
            ]
        }
    ]
}

And run the command below to apply this configuration:

$ aws s3api put-bucket-notification-configuration --bucket sauna-integration-test-bucket --notification-configuration file://snowplow-sauna-test-notification.json

Here's an example of a policy that can list/put/delete S3 objects and send/receive notifications to/from AWS SQS:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1471525597004",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:GetBucketNotification",
                "s3:PutBucketNotification"
            ],
            "Resource": [
                "arn:aws:s3:::sauna-integration-test-bucket/*"
            ]
        },
        {
            "Sid": "Stmt1471525597024",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::sauna-integration-test-bucket"
            ]
        },
        {
            "Sid": "Stmt1471525908000",
            "Effect": "Allow",
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage"
            ],
            "Resource": [
                "arn:aws:sqs:us-east-1:012345678912:sauna-integration-test-queue"
            ]
        },
        {
            "Sid": "Stmt1471525908001",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues"
            ],
            "Resource": [
                "arn:aws:sqs:us-east-1:012345678912:*"
            ]
        }
    ]
}

Don't forget to replace 012345678912 with your AWS Account ID and sauna-integration-test-queue and sauna-integration-test-bucket with SQS Queue and S3 Bucket respectively.