101. AWS EC2 01 - qyjohn/AWS_Tutorials GitHub Wiki

Setting up EC2 instances with SSH and RDP access. Using the AWS CLI, with a little bit on AWS credentials and IAM Roles. Getting started on AWS SDK for Python (boto3) and AWS SDK for Java.

(1) Launch EC2 Instances

Launch EC2 instances with various AMI's, and connect to the EC2 instances via either SSH or RDP. You should at least try Amazon Linux, Ubuntu, RHEL, and Windows. Use a t2.micro instance and terminate the instance when you are done with the exercise. The main purpose of this exercise is to get yourself familiar with the following topics:

Remember to terminate your EC2 instance when you are done with each experiment.

(2) Understand User Data and Metadata

Now, go through the following tutorials to understand metadata and user data. Please use a t2.micro instance with an Amazon Linux AMI for this exercise.

Remember to terminate your EC2 instance when you are done with each experiment.

(3) Using the AWS CLI

Starting from this step, we will be using Ubuntu 16.04 for the majority of our experiments. Launch a t2.micro instance with Ubuntu Server 16.04 LTS AMI. SSH into the EC2 instance and do the following:

$ sudo apt-get update
$ sudo apt-get install python-pip
$ sudo pip install awscli
$ aws --version
aws-cli/1.11.55 Python/2.7.12 Linux/4.4.0-64-generic botocore/1.5.18

Now you have the AWS CLI on your EC2 instance. Let's try to use the AWS CLI to describe do some simple things. Please remember to replace the instance-id and AWS region id in your "aws ec2 describe-instances" command.

$ curl http://169.254.169.254/latest/meta-data/instance-id
i-xxxxxxxxxxxxxxxxx

$ aws ec2 describe-instances --instance-ids i-xxxxxxxxxxxxxxxxx
You must specify a region. You can also configure your region by running "aws configure".

$ aws ec2 describe-instances --instance-ids i-xxxxxxxxxxxxxxxxx --region ap-southeast-2
Unable to locate credentials. You can configure credentials by running "aws configure".
You must specify a region. You can also configure your region by running "aws configure".

As you can see, the AWS CLI needs a set of AWS credentials to work. We recommend that you read the following AWS documentations before proceeding to the next steps.

Now, please create a new IAM user with the access type being "Programmatic access". In the "set permissions" step, choose "attach existing policies directly" and select "PowerUserAccess" for your exercises. Make sure to download your access keys and save it in a secure place. Remember that you do not share your access keys with anybody in any circumstances.

Run the following command to configure AWS CLI:

$ aws configure
$ aws ec2 describe-instances --instance-ids i-xxxxxxxxxxxxxxxxx

Use the --debug option with the AWS CLI to understand what happens during the API call. In particular, look into the time stamps to understand how much time is spent in each step. When is the API call being sent to the EC2 service endpoint? When is the HTTP response being received by the AWS CLI?

$ aws ec2 describe-instances --instance-ids i-xxxxxxxxxxxxxxxxx --debug

Read the following AWS CLI documentations, and get yourself familiar with the usage of some frequently used commands by running these commands multiple times. Notice the behavior difference between a reboot and a stop/start action. If the EC2 instance is using an automatically assigned public IP address, a reboot does not change the public IP address, while a stop/start changes the public IP address. At this point, you should get yourself familiar with the concept of Elastic IP Addresses (EIP), and use an EIP for your EC2 instance.

Create an AMI from the running instance, then launch a new instance with the newly created AMI. When you attempt to use the newly created AMI to launch a new instance, you may have noticed (through the describe-images command) that the newly created AMI does not immediately become available for use. This is because the API calls to create, modify, delete AWS resources are handled in an asynchronous way. The EC2 services endpoint returns "200 OK" immediately, while sending the API call to the background for processing. If you want to create a bash script to automate the process of creating an AMI and launching a new EC2 instance with the newly created AMI, you might want to use a loop to describe-images until the newly created AMI becomes available. This is going to be tricky, but fortunately the AWS CLI also provides a wait command:

Now, write two bash scripts to create an AMI for an EC2 instance, then launch a new EC2 instance from the newly created AMI. The bash script should be running inside the EC2 instance from which we are creating the AMI. The first bash script should use a loop of describe-images to wait for the AMI to become available. The second bash script should use the wait command to wait for the AMI to become available.

(4) Using IAM Role

Run the following commands in your EC2 instance, you will notice a hidden folder called ".aws". In this folder, your AWS credentials (access key and secrete key) are stored in a text file, in clear text. This is a security concern, because if a person has SSH access to the EC2 instance then he/she can easily copy your AWS credentials away. If you create an AMI and share your AMI with others, you might leak your AWS credentials without knowing that.

$ cd ~
$ ls
$ ls -a
$ cd .aws
$ ls
$ more credentials

The solution is to associate an IAM Role to your EC2 instance instead of using a set of access key and secret key in clear text. Please refer to the following AWS documentation to create an IAM Role for EC2, then associate the newly created IAM Role to your EC2 instance.

Amazon EC2 uses an instance profile as a container for an IAM role. When you create an IAM role using the IAM console, the console creates an instance profile automatically and gives it the same name as the role to which it corresponds. If you use the Amazon EC2 console to launch an instance with an IAM role or to attach an IAM role to an instance, you choose the instance profile based on a list of instance profile names.

If you use the AWS CLI, API, or an AWS SDK to create a role, you create the role and instance profile as separate actions, with potentially different names. If you then use the AWS CLI, API, or an AWS SDK to launch an instance with an IAM role or to attach an IAM role to an instance, you need to specify the instance profile name.

In the past, we can only associate an IAM Role to an EC2 instance when we launch the EC2 instance. Starting from February 2017, we can attach IAM Role to an existing EC2 instance.

But now you have two set of AWS credentials on your EC2 instance. One is the access key and secret key you specified using the "aws configure" command, the other is the IAM Role. How does the AWS CLI know which set of AWS credentials to use?

Run the following command and look into the debug information. It looks like that the AWS CLI is still using the access key and secret key you provided earlier.

$ aws ec2 describe-images --image-ids ami-xxxxxxxx --debug
......
2017-03-01 23:43:09,271 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2017-03-01 23:43:09,271 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2017-03-01 23:43:09,271 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2017-03-01 23:43:09,271 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
......

The AWS CLI looks for credentials and configuration settings in the following order:

  • Command Line Options – region, output format and profile can be specified as command options to override default settings.
  • Environment Variables – AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, etc.
  • The AWS credentials file – located at ~/.aws/credentials on Linux, macOS, or Unix, or at C:\Users\USERNAME .aws\credentials on Windows. This file can contain multiple named profiles in addition to a default profile.
  • The CLI configuration file – typically located at ~/.aws/config on Linux, macOS, or Unix, or at C:\Users\USERNAME .aws\config on Windows. This file can contain a default profile, named profiles, and CLI specific configuration parameters for each.
  • Instance profile credentials – these credentials can be used on EC2 instances with an assigned instance role, and are delivered through the Amazon EC2 metadata service.

Let's remove your credential file and try again. Now you can see that the AWS CLI makes two HTTP requests to obtain the AWS credentials - the first one to retrieve the name of the IAM Role, the second one to retrieve the actual AWS credentials information.

$ cd ~/.aws
$ rm credentials
$ aws ec2 describe-images --image-ids ami-xxxxxxxx --debug
......
2017-03-01 23:46:06,950 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2017-03-01 23:46:06,950 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2017-03-01 23:46:06,950 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2017-03-01 23:46:06,950 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2017-03-01 23:46:06,950 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2017-03-01 23:46:06,951 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2017-03-01 23:46:06,951 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: container-role
2017-03-01 23:46:06,951 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role
2017-03-01 23:46:06,955 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2017-03-01 23:46:06,956 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 10
2017-03-01 23:46:06,957 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2017-03-01 23:46:06,958 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/Test-EC2-Role HTTP/1.1" 200 906
2017-03-01 23:46:06,959 - MainThread - botocore.credentials - DEBUG - Found credentials from IAM Role: Test-EC2-Role
......

Now, remove the IAM Role from the EC2 instance using the EC2 Console. This can be done by selecting "No Role" on the "Attach/Replace IAM role" screen. Run the above-mentioned command again, you will see that although you still have the name of the IAM Role, but the AWS credentials are no longer there.

Associate and dis-associate IAM Role to your EC2 instance, then use the following commands to observe the behavior changes.

$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/[Name of the IAM Role]

(5) Some Python

Now we will use Python to interact with the EC2 service. It is OK if you have no idea about what Python is. We will learn this slowly.

First we install the AWS SDK for Python (boto3).

$ sudo pip install boto3

Then type "python" and press ENTER to get started.

$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> client = boto3.client('ec2', 'us-east-1')
>>> response = client.describe_instances()
>>> response
>>> exit()

The huge block of information from the last statement might look scary. It is in JSON format and you can read more about the request and response syntax from the following documentation:

If you want to read the output, we recommend that you copy the output and paste it into a "JSON beautifier" such as JSONLint to make it look pretty. In JSONLint, paste the output to the text field and click on "Validate JSON" and you will see the output in pretty format. JSONLint will also tell you that there are errors in the JSON data you paste, and you should simply ignore the error message. This is because of the The u- prefix added by Python to some of the strings in the JSON data, which means that you have a Unicode string. We recommend that you read the output in JSONLint and compare the output with the above-mentioned documentation. This will enable you to identify information blocks in the response from an API call.

Now, let's write some simply Python code:

import boto3

aws_region='ap-southeast-2'
client = boto3.client('ec2', aws_region)
response = client.describe_instances()
reservations = response['Reservations']
for reservation in reservations:
	print reservation['ReservationId']

Save the above-mentioned code block as test.py and execute it with the following command.

$ python test.py
r-0e0693190e19a8b05
r-07bd3bc7bee668904
r-017ec886db190ccaf
r-0c95a6176365f2ee6
r-7aa5f7b5
r-05aac06c08219b86e
r-0a5115d59a9b747c9
r-0388e6e951fa55b38
r-0ce499d203238529f
r-2a009af4
r-68010fb4
r-050de789729ce1ca6

In this exercise, you will see one of the most mysterious concept in EC2 - reservation. This does not mean EC2 Reserved Instancess (RI), but rather something else. A reservation is an act of launching instances. Basically, a reservation is what you do, while an instance is what you get. If you launch multiple instances from one image via run_instances() you make one reservation, but get multiple instances.

Now we improve our test.py with a print_instances() method to print out all the EC2 instances in a region:

import boto3

def print_instances(response):
	reservations = response['Reservations']
	for reservation in reservations:
		print reservation['ReservationId']
		instances = reservation['Instances']
		for instance in instances:
			print "\t", instance['InstanceId'], "\t", instance['State']['Name']
	return;

aws_region='ap-southeast-2'
client = boto3.client('ec2', aws_region)
response = client.describe_instances()
print_instances(response)

Using the EC2 Console, launch several EC2 instances with different number of instances for each launch. Then run the above-mention python program to see the output. Then try to stop or terminate some of the EC2 instances using the EC2 Console. After that, please modify the Python program to only print out EC2 instance in running, stopped, or terminated state.

$ python test.py
r-0e0693190e19a8b05
	i-0cb4e5013d1d94621 	running
r-07bd3bc7bee668904
	i-0f10d9e0bd742b5c8 	running
r-017ec886db190ccaf
	i-05a0199c5cc060e10 	running
r-0c95a6176365f2ee6
	i-033a2bd87d8772194 	running
	i-01f403d18a05ac142 	running
r-7aa5f7b5
	i-81c2565d 	running
r-05aac06c08219b86e
	i-082b699d1ceda63c7 	running
r-0a5115d59a9b747c9
	i-0c19f2fad86fe7bcb 	running
r-0388e6e951fa55b38
	i-0a9ad30e4aef67e16 	running
r-0ce499d203238529f
	i-00d7924a1bced1313 	running
r-2a009af4
	i-5e291480 	stopped
r-68010fb4
	i-248be6fb 	stopped
r-050de789729ce1ca6
	i-0c3d4969c77cfd7ea 	running
	i-02ede7ad4c4c4a463 	running

Now we add a single line to our test.py to enable logging for API calls to AWS, as below:

import boto3
boto3.set_stream_logger(name='botocore')

def print_instances(response):
	reservations = response['Reservations']
	for reservation in reservations:
		print reservation['ReservationId']
		instances = reservation['Instances']
		for instance in instances:
			print "\t", instance['InstanceId'], "\t", instance['State']['Name']
	return;

aws_region='ap-southeast-2'
client = boto3.client('ec2', aws_region)
response = client.describe_instances()
print_instances(response)

Run this code again, and you will see everything that happens in the botocore layer.

(6) Some Java

Now we will use Java to interact with the EC2 service. Again, it is OK if you have no idea about what Java is. We will learn this slowly.

First we will need to install the Java Development Kit (JDK) from Oracle:

$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt update; sudo apt install oracle-java8-installer
$ javac -version
javac 1.8.0_121
$ java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

Next we will need to download and install the AWS SDK for Java:

$ cd ~
$ wget https://sdk-for-java.amazonwebservices.com/latest/aws-java-sdk.zip
$ sudo apt-get install zip
$ unzip aws-java-sdk.zip

Depending on when you download the AWS SDK for Java, you will get a different version number when you unzip aws-java-sdk.zip. In my case I have aws-java-sdk-1.11.98. Now we will need to set the CLASSPATH environment variable for your Java application to work. It is recommended that you put this CLASSPATH definition into your .bashrc so that you do not need to do this every time you open a new SSH connection to the instance.

$ export CLASSPATH=~/aws-java-sdk-1.11.98/lib/aws-java-sdk-1.11.98.jar:~/aws-java-sdk-1.11.98/third-party/lib/*:.

In the above-mentioned CLASSPATH definition, we use a wildcard * to include all the JAR files under a directory. This is not a well known feature that was introduced in Java 1.6 but you should know about it. Also, we recommend that you read the following Oracle documentation on CLASSPATH to understand how it works.

Now, let's write some simple Java code:

import java.util.*;
import com.amazonaws.*;
import com.amazonaws.auth.*;
import com.amazonaws.auth.profile.*;
import com.amazonaws.regions.*;
import com.amazonaws.services.ec2.*;
import com.amazonaws.services.ec2.model.*;

public class TestEC2 
{
        public AmazonEC2Client client;

        public TestEC2()
        {
                client = new AmazonEC2Client();
                client.configureRegion(Regions.AP_SOUTHEAST_2);
        }

        public void listInstances()
        {
                try 
                {
                        DescribeInstancesResult result = client.describeInstances();
                        List<Reservation> reservations = result.getReservations();
                        for (Reservation reservation: reservations)
                        {
                                String reservation_id = reservation.getReservationId();
                                System.out.println("Reservation: " + reservation_id);
                                List<Instance> instances = reservation.getInstances();
                                for (Instance instance: instances)
                                {
                                        String id = instance.getInstanceId();
                                        String state = instance.getState().getName();
                                        System.out.println("\t" + id + "\t" + state);
                                }
                        }
                } catch (Exception e) 
                {
                        System.out.println(e.getMessage());
                        e.printStackTrace();
                }
        }

        public static void main(String[] args)
        {
                TestEC2 test = new TestEC2();
                test.listInstances();
        }
}

Save this code as TestEC2.java, then compile and run it with the following commands. Similar to what we have done with our Python code, this Java code prints out all the reservations and the EC2 instances for each reservation in a region.

$ javac TestEC2.java
$ java TestEC2
Reservation: r-0e0693190e19a8b05
	i-0cb4e5013d1d94621	running
Reservation: r-07bd3bc7bee668904
	i-0f10d9e0bd742b5c8	running
Reservation: r-017ec886db190ccaf
	i-05a0199c5cc060e10	running
Reservation: r-0c95a6176365f2ee6
	i-033a2bd87d8772194	running
	i-01f403d18a05ac142	running
Reservation: r-7aa5f7b5
	i-81c2565d	running
Reservation: r-05aac06c08219b86e
	i-082b699d1ceda63c7	running
Reservation: r-0a5115d59a9b747c9
	i-0c19f2fad86fe7bcb	running
Reservation: r-0388e6e951fa55b38
	i-0a9ad30e4aef67e16	running
Reservation: r-0ce499d203238529f
	i-00d7924a1bced1313	running
Reservation: r-2a009af4
	i-5e291480	stopped
Reservation: r-68010fb4
	i-248be6fb	stopped
Reservation: r-050de789729ce1ca6
	i-0c3d4969c77cfd7ea	running
	i-02ede7ad4c4c4a463	running

When we develop applications we have a need to enable logging for debug purposes. The AWS SDK for Java is instrumented with Apache Commons Logging, which is an abstraction layer that enables the use of any one of several logging systems at runtime. We do not need to make any code changes to enable logging API calls to AWS. All we need to download log4j from a mirror, and set up the appropriate CLASSPATH:

$ cd ~
$ wget http://apache.mirror.digitalpacific.com.au/logging/log4j/1.2.17/log4j-1.2.17.zip
$ unzip log4j-1.2.17.zip
$ export CLASSPATH=$CLASSPATH:~/apache-log4j-1.2.17/log4j-1.2.17.jar

Then we will need to have a file called log4j.properties in the same folder as your TestEC2.java and TestEC2.class, with the following content:

log4j.rootLogger=WARN, A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c -  %m%n
# Turn on DEBUG logging in com.amazonaws.request to log
# a summary of requests/responses with AWS request IDs
log4j.logger.com.amazonaws.request=DEBUG

Now we run our TestEC2 again, we can see how the API call is being sent to the EC2 service endpoint, and how the EC2 service endpoint returns information such as HTTP status code and AWS Request ID.

$ java TestEC2 
2017-03-02 09:34:30,158 [main] DEBUG com.amazonaws.request -  Sending Request: POST https://ec2.ap-southeast-2.amazonaws.com / Parameters: ({"Action":["DescribeInstances"],"Version":["2016-11-15"]}Headers: (User-Agent: aws-sdk-java/1.11.98 Linux/4.4.0-64-generic Java_HotSpot(TM)_64-Bit_Server_VM/25.121-b13/1.8.0_121, amz-sdk-invocation-id: 7af93fad-9725-42b3-58d3-31352c5ef784, ) 
2017-03-02 09:34:32,008 [main] DEBUG com.amazonaws.request -  Received successful response: 200, AWS Request ID: 92415d75-d580-44d8-9299-a756304ab6e6
2017-03-02 09:34:32,009 [main] DEBUG com.amazonaws.request -  x-amzn-RequestId: not available
2017-03-02 09:34:32,009 [main] DEBUG com.amazonaws.request -  AWS Request ID: 92415d75-d580-44d8-9299-a756304ab6e6
Reservation: r-0e0693190e19a8b05
	i-0cb4e5013d1d94621	running
Reservation: r-07bd3bc7bee668904
	i-0f10d9e0bd742b5c8	running
Reservation: r-017ec886db190ccaf
	i-05a0199c5cc060e10	running
Reservation: r-0c95a6176365f2ee6
	i-033a2bd87d8772194	running
	i-01f403d18a05ac142	running
Reservation: r-7aa5f7b5
	i-81c2565d	running
Reservation: r-05aac06c08219b86e
	i-082b699d1ceda63c7	running
Reservation: r-0a5115d59a9b747c9
	i-0c19f2fad86fe7bcb	running
Reservation: r-0388e6e951fa55b38
	i-0a9ad30e4aef67e16	running
Reservation: r-0ce499d203238529f
	i-00d7924a1bced1313	running
Reservation: r-2a009af4
	i-5e291480	stopped
Reservation: r-68010fb4
	i-248be6fb	stopped
Reservation: r-050de789729ce1ca6
	i-0c3d4969c77cfd7ea	running
	i-02ede7ad4c4c4a463	running

For more information on logging for the AWS SDK for Java, we recommend that you read through the following AWS documentation:

Also, you should start learning how to read Java docs:

(7) Summary and Homework

In this tutorial, we learn some basic skill on how to use the AWS EC2 service using the AWS EC2 Console, the AWS CLI, the AWS SDK for Python (boto3), and the AWS SDK for Java. As you can see, once you know how to do all these they are in fact quite simple!

As your homework, you will need to develop a bash script, a Python script, and a Java application to create an AMI from a running EC2 instance, wait for the AMI to be available, launch an EC2 instance with the newly created AMI, wait for the newly launched EC2 instance to pass 2/2 health checks, then terminate the newly launched EC2 instance. Oh yes your code needs to be running from within the EC2 instance from which you will make the new AMI. :-)

Keep one EC2 instance with all the above-mention software, SDK, configuration as your development environment. Terminate all other EC2 instances when you are done with the exercises.

Have a great day!

⚠️ **GitHub.com Fallback** ⚠️