Ubuntu as AWS Spot - hpaluch/hpaluch.github.io GitHub Wiki

Ubuntu on AWS as m1.small Spot instance for $0.0044/hr

This page was created on May 2016 and updated first in October 2016 and later updated on August 2018

Here is quick guide how to request Ubuntu Linux as m1.small spot instance on AWS for just $0.0044/hr.

Current (August 2018) m1.small On-Demand price for Linux is (from https://aws.amazon.com/ec2/previous-generation/):

Type vCPU ECU Memory (GiB) Instance Storage (GB) Price
m1.small 1 1 1.7 1 x 160 $0.044 per Hour

Typical spot price for us-east-1c (August 2018) is:

AWS Spot prices for m1.small classic

NOTE: Good news: The Spot price declined from 0.0071 (October 2016) to 0.0044 (August 2018).

Requirements

  • you need to have installed and working AWS CLI
  • you need proper permissions for your AWS CLI user
  • you need to have created Key-pair to access your Linux instance (named as "your_keypair" in this example) and you need to have this keypair accessible on your ssh client machine
  • you need to have created Security Group - at least port 22 for SSH should be enabled there

Example permissions for AWS CLI user

Here are my example permissions (set them via IAM console):

Managed Policies:

  • AmazonEC2ReadOnlyAccess
  • AmazonVPCReadOnlyAccess

Inline Policies:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1462180585000",
            "Effect": "Allow",
            "Action": [
                "ec2:StopInstances",
                "ec2:TerminateInstances"
            ],
            "Resource": [
                "arn:aws:ec2:us-east-1::instance/*"
            ]
        },
        {
            "Sid": "Stmt1462180585001",
            "Effect": "Allow",
            "Action": [
                "ec2:RequestSpotInstances"
            ],
            "Resource": [
                "*"
            ],
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": "X.X.X.X/32"
                }
            }
         }
    ]
}

NOTE: replace X.X.X.X on line "aws:SourceIp": "X.X.X.X/32" with public IP address of your Client Linux (where you run AWS CLI commands).

You can invoke this command on your Client Linux to get your Public IP address:

curl  http://checkip.amazonaws.com

Please refine and review these policies before applying them.

Keep in mind to never expose access keys of such powerful IAM users - malicious people can misuse it to create hundred of Spot instances for each AZ thus having serious consequences for you!

See https://bitcalm.com/blog/howi-became-12000-amazons-debtor-in-24-hours/ for one such example (with happy end).

Setup scripts

Create request_spot_ubuntu.json with this contents - do not forget to replace:

  • your_keypair with your Key-pair name
  • sg-12345678 with your security group
  • you can leave ami-06e5b20beb4497111 unchanged for Ubuntu 16.04 LTS 20190617, 64-bit
  • you can change Placement/AvailabilityZone to reflect best price

File request_spot_ubuntu.json:

{
  "ImageId": "ami-06e5b20beb4497111",
  "KeyName": "your_keypair",
  "SecurityGroupIds": [ "sg-12345678" ],
  "InstanceType": "m1.small",
  "Placement": {
    "AvailabilityZone": "us-east-1c"
  }
}

If you want another Ubuntu edition (for example 64bit) you can choose from list at http://cloud-images.ubuntu.com/locator/ec2/

However keep in mind that for m1.small it must be:

  • Region: us-east-1
  • Storage: instance-store (not hvm-instance)
  • do NOT use VPC - I have no luck fulfilling Spot requests for m1.small in VPC (why?)

Another note: if you choose improper image the Spot request may return confusing errors (like Capacity not available)

  • create shell script request_spot_ubuntu.sh with contents:
et -ex
cd `dirname $0`
dryrun=" --dry-run "

aws ec2 request-spot-instances $dryrun --spot-price 0.0049 \
   --instance-count 1 \
   --type one-time \
   --launch-specification file://`pwd`/request_spot_ubuntu.json
  • fix permissions on script:
chmod a+rx request_spot_ubuntu.sh
  • try this script (it is in dry-run mode):
./request_spot_ubuntu.sh
  • if everything works well, comment out the line
#dryrun=" --dry-run "

  • and run that script again

  • Watch your request state and/or running instances on AWS console

Connecting to running Ubuntu instance

Install jq to your client (workstation) Linux, for example:

sudo apt-get install jq

Use this command to get Public DNS names of all your EC2 Spot Instances:

aws ec2 describe-instances | jq '.Reservations[].Instances[] | select ( .InstanceLifecycle == "spot" ) | .PublicDnsName'
  OUTPUT: "ec2-1-2-3-4.compute-1.amazonaws.com"

Invoke this command (replace your_keypair.pem and YOUR_INSTANCE_PUBLIC_IP):

ssh -i path_to_your/your_keypair.pem ubuntu@YOUR_INSTANCE_PUBLIC_IP

You can upgrade your Ubuntu to latest version using command:

sudo apt-get upgrade

NOTE: you may be billed additional "inter AZ data transfer rate" (!) - about $0.01 for 300MB of upgrade data

Fix Invalid locale problem

While using Ubuntu 16.04.5 LTS (AMI ID: ami-06e5b20beb4497111) I encountered very strange locale errors on them:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LC_TIME = "cs_CZ.UTF-8",
        LC_MONETARY = "cs_CZ.UTF-8",

I was really puzzled - how ever my Czech locale settings can appear on remote Ubuntu instance? The solution is easy https://stackoverflow.com/a/12024385 - reconfigure your workstation (= ssh client) to NOT send locale variables to remote server (!).

Real surprise :-)

Billing overview

Currently best billing view is: