6.3 Create an end-to-end continuous compliance solution using CloudFormation and CodePipeline

Review and ensure that you have setup your development environment before going through the steps below.

In this sublesson, you will walk through an exercise to deploy a full-stack and fully-automated Continuous Compliance solution on AWS using Amazon CloudWatch Event Rules, AWS Config and Config Rules, IAM, AWS CodePipeline, AWS CodeBuild, AWS Lambda, Amazon SNS, Amazon S3, cfn_nag, and other tools and services.

Disable AWS Config

Replace ceoa-***, CONFIGRECORDERNAME, and DELIVERYCHANNELNAME with the actual name of the resources.

aws configservice delete-config-rule --config-rule-name s3-bucket-server-side-encryption-enabled
aws configservice describe-configuration-recorders
aws configservice delete-configuration-recorder --configuration-recorder-name default
aws configservice describe-delivery-channels
aws configservice delete-delivery-channel --delivery-channel-name default

Deployment Steps

There are four main steps in launching this solution: preparing an AWS account, create and store source files, launching the CloudFormation stack, and testing the deployment. Each is described in more detail in this section. Please note that you are responsible for any fees incurred while creating and launching your solution.

Step 1. Prepare an AWS Account

Here are the prerequisites and assumptions for this solution:

• AWS Account – Follow these instructions to create an AWS Account: Creating an AWS Account. Be sure you have signed up for the CloudFormation service and have the proper permissions.
• AWS CloudTrail – A CloudTrail log should already be enabled for the region in which you are running this example. See Creating a Trail for more information.
• AWS Region – Use the region selector in the navigation bar of the console to choose your preferred AWS region.
• AWS Cloud9 – All of the CLI examples assume that you are using the AWS Cloud9 IDE. To learn how to install and configure Cloud9, go to Automating AWS Cloud9. You can easily adapt the commands to perform the same actions in a different IDE, but the instructions are outside the scope of this sublesson.
• AWS Config – This solution assumes you have not enabled AWS Config. If you have, you will receive an error when launching the CloudFormation stack as it provisions a Config Recorder and Delivery Channel.

Step 2. Create and Store Source Files

In this section, you will create source files that will be stored in S3 and then uploaded to AWS CodeCommit when launching the CloudFormation stack. The names are listed below.

From your AWS Cloud9 terminal, type the following to setup your directory structure:

cd ~/environment/ccoa
aws s3 mb s3://ccoa-6-$(aws sts get-caller-identity --output text --query 'Account')
mkdir codecommit-files
cd ~/environment/ccoa/codecommit-files

Create empty source files:

touch buildspec.yml
touch buildspec-lambda.yml
touch ccoa-6-pipeline.yml
touch index.js
touch package.json
touch README.md
touch sam-s3-remediation.yml
touch volume.yml

Save the files.

buildspec.yml

Copy the contents from buildspec.yml to your local buildspec.yml file in Cloud9 and save it.

buildspec-lambda.yml

Copy the contents from buildspec-lambda.yml to your local buildspec-lambda.yml file in Cloud9 and save it.

AWS CodeBuild will use this buildspec to build the Lambda function that runs the automatic compliance remediation to fix the S3 Bucket with a bucket that is too permissive.

CodeBuild runs an aws cloudformation CLI command to package a SAM template and then exports the contents as a zip file into S3 so that Lambda can run the code.

ccoa-6-pipeline.yml

Copy the source contents from the ccoa-6-pipeline.yml file and save it to your local file in your Cloud9 environment called ccoa-6-pipeline.yml. The file is a 500-line CloudFormation template.

index.js

Copy the source contents from the index.js file and save it to your local file in your Cloud9 environment called index.js. This is the Node.js function that Lambda runs to remove the S3 bucket policy for S3 buckets that allow writes to their buckets.

package.json

Copy the source contents from the package.json file and save it to your local file in your Cloud9 environment called package.json. This is a metadata file that all Node.js apps need to operate.

sam-s3-remediation.yml

Copy the source contents from the sam-s3-remediation.yml file and save it to your local file in your Cloud9 environment called sam-s3-remediation.yml. This is the SAM template that defines the Lambda function and the CloudWatch Event Rule that triggers the Lambda function. The purpose of this CloudWatch Event Rule is to detect noncompliant S3 buckets and then automatically run a Lambda function that remediates these noncompliant buckets.

volume.yml

Copy the contents from volume.yml to your local volume.yml file in Cloud9 and save it.

Sync the files with your S3 bucket

In this section, you will zip and upload all of the source files to S3 so that they can be committed to the CodeCommit repository that is automatically provisioned by the stack generated by the ccoa-6-pipeline.yml template.

NOTE: Make a note of the S3 bucket you create, because you will use this as a parameter when launching your CloudFormation stack.

From your AWS Cloud9 environment, type the following:

cd ~/environment/ccoa/codecommit-files
zip ccoa-lesson6-examples.zip *.*
aws s3 sync ~/environment/ccoa/codecommit-files s3://ccoa-6-$(aws sts get-caller-identity --output text --query 'Account')

Step 3. Launch the Stack

From your AWS Cloud9 environment, type the following (replacing [email protected] with the appropriate value):

aws cloudformation create-stack --stack-name ccoa-6-ar --template-body file:///home/ec2-user/environment/ccoa/codecommit-files/ccoa-6-pipeline.yml --parameters ParameterKey=EmailAddress,[email protected] ParameterKey=CodeCommitS3Bucket,ParameterValue=ccoa-6-$(aws sts get-caller-identity --output text --query 'Account') ParameterKey=CodeCommitS3Key,ParameterValue=ccoa-lesson6-examples.zip ParameterKey= CreateConfig,ParameterValue=create --capabilities CAPABILITY_NAMED_IAM --disable-rollback

Step 4. Test the Deployment

First, verify that the CloudFormation stack you just launched (called ccoa-6-ar) was successfully created. Click on the PipelineUrl Output to launch deployment pipeline in CodePipeline to see it running. Verify that the pipeline successfully went through all stages (as shown below).

Next, you will create an S3 bucket that allows people to put files into the bucket. We are doing this for demonstration purposes since you should not grant any kind of public write access to your S3 bucket. Here are the steps:

1. Go to your Cloud9 IDE terminal and type the following: aws s3 mb s3://ccoa-6-s3-violation-$(aws sts get-caller-identity --output text --query 'Account') --region $(aws configure get region --output text)
2. Go to the S3 console and select the ccoa-6-s3-violation-ACCOUNTID bucket and choose the Permissions pane.
3. Click on Bucket Policy and paste the contents from s3-bucket-policy.json into the Bucket policy editor text area. IMPORTANT: Be sure to replace ACCOUNTID with your AWS Account Id.
4. Click the Save button.

You will receive this message:

You have provided public access to this bucket. We highly recommend that you never grant any kind of public access to your S3 bucket.

Verify Compliance

In this section, you will verify that the Config Rule has been triggered and that the S3 bucket resource has been automatically remediated.

1. Go to the Config console.
2. Click on Rules.
3. Select the s3-bucket-public-write-prohibited rule.
4. Click the Re-evaluate button.
5. Go back to Rules in the Config console.
6. Go to the S3 console and choose the ccoa-6-s3-violation-ACCOUNTID bucket (replacing ACCOUNTID with your AWS account id).
7. Click on the Permissions pane.
8. Click on the Bucket Policy and ensure that bucket policy has been removed.
9. Go back to Rules in the Config console and confirm that the s3-bucket-public-write-prohibited rule is Compliant.

What's Next?

In this sublesson, you learned how to setup a robust automated compliance and remediation infrastructure for noncompliant AWS resources using services such as S3, AWS Config & Config Rules, Amazon CloudWatch Event Rules, AWS Lambda, IAM, and others. You did this by automating all of the provisioning by using tools like AWS CloudFormation, AWS CodePipeline, AWS CodeCommit, and AWS CodeBuild.

By leveraging this approach, your AWS infrastructure is capable of rapidly scaling resources while ensuring these resources are always in compliance without humans needing to manually intervene.

Consider the possibilities when adding hundreds if not thousands of rules and remediations to your AWS infrastructure. Below is just an example of some of the different types of Managed Config Rules you can run. What if you took each of these and developed custom remediations for them and ensured they were running across all of your AWS accounts? Or, what if you wrote your own Config Rules and triggered CloudWatch Events to execute remediations you developed in Lambda? This way your compliance remains in lockstep with the rest of your AWS infrastructure.

As a result, engineers can focus their efforts on automating the prevention, detection, and remediation of their AWS infrastructure rather than manually hunting down every noncompliant resource, creating a ticket, and manually fixing the issue. This is Continuous Compliance – at scale!