Amazon Web Services - BenLangmead/jhu-compute GitHub Wiki
These instructions are outdated. There was a costly breach of the lab's AWS account in June 2015 and we need to reconsider security policies. If you have a need for AWS please talk to Ben.
People in the lab are sometimes given Amazon Web Services account credentials (and EC2 credentials, and other related credentials) so that they can use AWS resources for research activities.
Setup
To set up such an account, see Ben. Toegether, we'll:
- Make a user in the langmead lab group.
- Transfer your AWS Access Key ID and AWS Secret Key to you. Be careful with the AWS Secret Key; that is for your eyes only.
- Make a password, which you will use to log into the various AWS web interfaces, including the AWS Console.
- Transfer any relevant EC2 keypairs to you.
Usually, all transfers of sensitive information will be via the HHPC cluster. I.e. Ben will make a credential, transfer it to HHPC temporarily, then you will log into HHPC and copy it to your computer. Ben will then delete all copies.
Other software you should install locally
- The AWS CLI (install e.g. with
conda install awscliorpip install awscli), which is a complete suite of command-line tools for interacting with AWS. It's generally preferable to the tools described below. s3cmd, which allows you to interact with S3 via the command line- The
elastic-mapreduceruby script, which allows you to launch Elastic MapReduce clusters from the command line. It’s used by Crossbow and Myrna. boto, a Python library for interacting with AWS
Lab login page
Start here to log in to the AWS console as a member of the lab.
How to use AWS (the very short version)
| If you want to ... | ...then use |
|---|---|
| Launch Elastic Compute Cloud (EC2) Instance(s) | AWS Management Console |
| Monitor (EC2) Instance(s) already launched | AWS Management Console |
| Launch Elastic MapReduce (EMR) cluster(s) | AWS Management Console |
| Monitor (EMR) cluster(s) already launched | AWS Management Console |
| Browse S3 Buckets | AWS Management Console (or s3cmd, Cyberduck, s3fox, etc) |
| Transfer files to/from S3 | AWS Management Console (or s3cmd, Cyberduck, s3fox, etc) |
Be careful
Do not leave clusters running for (much) longer than necessary! Leaving a big cluster running for a long time is a great way to accidentally spend thousands of the laboratory’s dollars. Once your cluster is launched, you should constantly feel a creeping paranoia that you’re going to forget to terminate it.
Billing
Anyone with an account can check on the lab's money situation on the AWS billing page. Usually we are using "credits" from grants awarded by Amazon. Once we use up the credits, then we have to start paying real money. Please let Ben know immediately if you see anything surprising or alarming about the billing.