Setup IAM permissions for users installing Snowplow - OXYGEN-MARKET/oxygen-market.github.io GitHub Wiki

Setting up permissions in IAM for the user(s) installing Snowplow is an 3 step process:

• 1 Create an IAM group (incl. creating a user and setting permissions)
  • 1.1 Batch Permissions
  • 1.2 Kinesis Permissions
  • 1.3 Kinesis & Batch
• 2 Enable users to log into AWS

Disclaimer: Snowplow Analytics Ltd will not be liable for any problems caused by the full or partial implementation of these instructions on your Amazon Web Services account. If in doubt, please consult an independent AWS security expert.

Warning: these permissions are still more permissive than they need to be. We will be putting in time to narrow them down further over the coming weeks.

First click on the IAM icon on the AWS dashboard:

Now click on the Create a New Group of Users button:

Enter a Group Name of snowplow-setup:

Now choose the Custom Policy option and click Select:

Let's give it a Policy Name of snowplow-policy-setup-infrastructure:

Depending on what Pipeline needs to be setup you will need slightly varying permissions.

The following permissions are needed for all batch proccessing operations:

• Amazon S3
• Amazon EMR
• Amazon EC2 (required for EmrEtlRunner / StorageLoader)
• Amazon Marketplaces (required for EmrEtlRunner / StorageLoader)
• Amazon CloudFront (required for Cloudfront collector)
• Amazon Elastic Beanstalk (required for Clojure collector)
• Amazon Elasticsearch service (for loading bad rows)
• Amazon Redshift (required for Redshift)
• Amazon Cloudformation (required if the Snowplow team setup your Snowplow data pipeline, as we use Cloudformation)
• Amazon IAM (required as part of the Clojure collector setup, as a role is created for the Clojure collector application)
• Amazon RDS (PostgreSQL server required by Iglu Server)

If you are not using the Clojure Collector, you can remove the Elastic Beanstalk section.

Paste the following JSON into the Policy Document text area:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "acm:*",
        "autoscaling:*",
        "aws-marketplace:ViewSubscriptions",
        "aws-marketplace:Subscribe",
        "aws-marketplace:Unsubscribe",
        "cloudformation:*",
        "cloudfront:*",
        "cloudwatch:*",
        "ec2:*",
        "elasticbeanstalk:*",
        "elasticloadbalancing:*",
        "elasticmapreduce:*",
        "es:*",
        "iam:*",
        "rds:*",
        "redshift:*",
        "s3:*",
        "sns:*"
      ],
      "Resource": "*"
    }
  ]
}

For the Kinesis Pipeline ending in an Elasticsearch Cluster you will need these permissions:

• Amazon EC2
• Amazon Cloudformation (required if the Snowplow team setup your Snowplow data pipeline, as we use Cloudformation)
• Amazon IAM (required as each Application for the Kinesis Pipeline has its own IAM Policy)
• Amazon Kinesis (required for Kinesis Streams to be created)
• Amazon DynamoDB (required for Kinesis Applications to work with Kinesis Streams)
• Amazon CloudWatch/Logs
• Amazon AutoScaling (required for all Kinesis Applications to work effectively)
• Amazon ElasticLoadBalancing (required for the Kinesis Collector)

Paste the following JSON into the Policy Document text area:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "acm:DescribeCertificate",
        "acm:ListCertificate",
        "autoscaling:*",
        "elasticloadbalancing:*",
        "kinesis:*",
        "iam:*",
        "cloudwatch:*",
        "ec2:*",
        "cloudformation:*",
        "cloudfront:*",
        "logs:*",
        "dynamodb:*",
        "sns:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

For the Kinesis Pipeline ending in the LZO S3 Sink with events from this sink then being processed in batches you will need these permissions:

• Amazon S3
• Amazon EMR
• Amazon EC2
• Amazon Marketplaces (required for EmrEtlRunner / StorageLoader)
• Amazon Redshift (required for Redshift)
• Amazon Cloudformation (required if the Snowplow team setup your Snowplow data pipeline, as we use Cloudformation)
• Amazon IAM (required as each Application for the Kinesis Pipeline has its own IAM Policy)
• Amazon RDS (PostgreSQL server required by Iglu Server)
• Amazon Kinesis (required for Kinesis Streams to be created)
• Amazon DynamoDB (required for Kinesis Applications to work with Kinesis Streams)
• Amazon CloudWatch/Logs
• Amazon AutoScaling (required for all Kinesis Applications to work effectively)
• Amazon ElasticLoadBalancing (required for the Kinesis Collector)

Paste the following JSON into the Policy Document text area:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "acm:*",
        "autoscaling:*",
        "aws-marketplace:Subscribe",
        "aws-marketplace:Unsubscribe",
        "aws-marketplace:ViewSubscriptions",
        "cloudformation:*",
        "cloudfront:*",
        "cloudwatch:*",
        "dynamodb:*",
        "ec2:*",
        "es:*",
        "elasticbeanstalk:*",
        "elasticloadbalancing:*",
        "elasticmapreduce:*",
        "iam:*",
        "kinesis:*",
        "logs:*",
        "rds:*",
        "redshift:*",
        "s3:*",
        "sns:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Once you have selected the correct Policy Document click Continue:

From the Add Existing Users tab, switch to the Create New Users tab:

Now enter a first User Name - we use snowplow-setup:

Keep the option Generate an access key for each User checked, and then click Continue.

Check that the configuration for your new IAM group looks something like this:

Click Continue and you should see the following:

Click Download Credentials to save these credentials locally. Then click Close Window.

Provide these credentials in a secure way - not via email - to whoever is setting up Snowplow for you, so that they can add them into the configuration of your EmrEtlRunner and StorageLoader applications.

Back to top.

For much of the Snowplow setup process, the IAM user you have setup above will need access to the Amazon Web Services control panel.

From within the Users tab inside the IAM dashboard, click on your snowplow user:

Now switch to the Security Credentials tab in the bottom pane, and click Manage Password on the right:

Now choose Assign an auto-generated password:

Click Apply and you should see the following:

Click Download Credentials to save these credentials locally. Then click Close Window.

Now, provide the following details in a secure way - not via email - to whoever is setting up Snowplow for you:

• Login URL: https://snplow.signin.aws.amazon.com/console
• Username: snowplow
• Password: as downloaded

Back to top.