AWS - nimrody/knowledgebase GitHub Wiki

• Cloudformation

Networking

• Networking on AWS and discussion on HN

IAM

• Policies

Monitoring

• Docker container for monitoring

New EC2 installation for java

sudo yum install tmux dstat htop
sudo yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel

sudo yum install tmux dstat htop  java-1.8.0-openjdk java-1.8.0-openjdk-devel

sudo alternatives --config java 

scp -r  /home/nimrody/dev/server/.aws drill:

java-1.8.0-openjdk-devel   ## for jstat

• Add disk (block device - lsblk)

  mkdir ~/a sudo mkfs -t ext4 /dev/xvdb sudo mount /dev/xvdb ~/a sudo chown ec2-user:ec2-user $HOME/a

Instance metadata

• List available metadata categories: curl http://169.254.169.254/latest/meta-data/

EC2 roles

• Can use InstanceProfileCredentialsProvider to automatically retrieve and refresh credentials from EC2 roles (see here

  AmazonS3 s3 = AmazonS3ClientBuilder.standard() .withCredentials(new InstanceProfileCredentialsProvider()) .build();

Monitoring

• Monitoring EC2 memory/disk/etc.

ECS - Container service

• ECS challenges

Misc.

• Sending SQS message using the command line

  aws --region=us-west-2 sqs send-message --queue-url=https://sqs.us-west-2.amazonaws.com/account/n_test_queue --message-body=xxx

  Or use --profile=... to set profile.

• s3-cli for fast parallel download/upload

• Cloudwatch logs agent

  See AWS docs on how to setup IAM policy that enables sending logs.

  Install the agent

  sudo yum update -y sudo yum install -y awslogs

  Configure the agent

  /etc/awslogs/awscli.conf - define API key / secret

  /etc/awslogs/awslogs.conf - define agent parameters - time format, polling period, filename

  /var/log/awslogs.log - agent logfile

  sudo service awslogs start - to start the agent

  sudo chkconfig awslogs on - to enable the service on boot

  Use ansible to install files

  ansible prodpolicy2 -s  -m copy -a "src=awslogs/awslogs.conf dest=/etc/awslogs/awslogs.conf"
  

  • Python library boto3

DNS

• Setup BIND to forward dns requests for internal address to AWS

Granting access to specific bucket

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::www.tensera.net",
                "arn:aws:s3:::www.tensera.net/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name/*"
            ]
        }
    ]
}