Section 6: Interacting with Other Computers & Virtual Terminal Sessions - Green-Biome-Institute/AWS GitHub Wiki

Back to Section 5: More Complex but Useful Commands

Learning Points for Section 6 - Interacting with Other Computers & Virtual Terminal Sessions

What you should take away from this section:

The SSH (Secure Shell Protocol) command allows for you to create a secure connection to another computer host (like an EC2 instance), giving you access to the information on the computer and ability to execute commands on it. SSH usually requires a password or keypair to create the connection.
The SCP (Secure Copy Protocol) command allows you to copy files to and from remote computer hosts (again, like an EC2 instance). SCP also requires a password or keypair for permission.
(Control + c) allows for you to force quit any process that is currently operating (for example if it is stuck and not ending or you realized you have to change something int the command you used)
tmux and screen can be used to create virtual terminals, which are CLI sessions operating behind the scenes where programs can run while you exit the EC2 instance (and say… go to sleep while they work!)

Secure Shell Protocol

`SSH`

The Secure Shell Protocol, SSH, is used to create a secure connection between your computer and a remote computer host. Before we jump into this, let’s look at an example. All you have to do is exit your EC2 instance. Then once logged out, press the up arrow once, which should reload the SSH command we originally used at the beginning of this module to log into our EC2 instance. This looks like:

$ ssh -i ~/Desktop/GBI/AWS/AWS_keypairs/GBI-training-module-keypair.pem [email protected]

Where all the x’s represent numbers that will be different for most of you. Breaking this down:

ssh is the command for SSH
-i is an option that tells SSH that the next file following this option will be “the identity (private key) for public key authentication is read.” In this situation there are 2 “keypairs”, one is called the public key and one is called the private key. The public key is stored by AWS and the private key is stored (and kept secret!) by us. Without the private key, you will not be given permission to create a connection to the remote host.
~/Desktop/GBI/AWS/AWS_keypairs/ is an example path on my own personal computer to a folder of AWS EC2 instance keypairs.
GBI-training-module-keypair.pem is the name of the private keypair for the EC2 instance we are using, within my keypair directory (this is the same keypair for all of the EC2 instances created for this tutorial).
ubuntu is the username of the EC2 instance
@ec2-xx-xx-xxx-xxx.us-west-2.compute.amazonaws.com is the hostname of the EC2 instance.

So now you can have a better understanding of this command. We are creating a secure connection between our computer and an EC2 instance with the address ec2-xx-xx-xxx-xxx.us-west-2.compute.amazonaws.com, logging in as the user ubuntu and passing the keypair (like entering the password to the EC2 instance) GBI-training-module-keypair.pem!

As long as you have a network connection to another computer (we are connected to AWS via the internet), know what kind of permission is required to access it, and have the correct keypair/password, you are able to connect with many different computers!

Secure Copy Protocol

`SCP`

The Secure Copy Protocol, SCP, builds upon SSH. Instead of just creating a secure connection to another computer host, however, it copies information either to or from it and your own computer! The syntax for this is very similar to SSH and cp, so we won’t go into such detail with the terms. The command syntax for copying from the remote computer host to your own computer looks like:

scp -[options] [keypair-if-necessary] [username]@[hostname]:[path/to/file/on/remote/computer/desired-file.filetype] [path/to/location/you/want/it/copied]

For copying from your computer to the remote computer host, just flip the last two terms, like you would do in the normal cp command.

Let’s do an example of this. I have a program on my EC2 instance that you will need to copy onto your own! Let’s figure out how to get that file.

To start, the keypair for my EC2 instance is the same as for yours. It’s the same file as you used in the SSH command to log onto your EC2 instance earlier in this section. I have placed a copy of this file in the ~/ex5-dir/ directory on your EC2 instance. You'll see two keypair files in there, so make sure to choose the right one. It is named GBI-Bioinformatics-Class.pem.sh. Therefore the path to this keypair file is:

~/extra/GBI-Bioinformatics-Class.pem.sh

Next, we need the path to the file on my computer. We can use the following information to create that path:

My username is also ubuntu
My hostname address is ec2-xx-xx-xxx-xxx.us-west-2.compute.amazonaws.com (I will give you this address in class. If you are doing this on your own you can skip this step, just try to understand the concept).
The path to the file I want you to copy from my EC2 instance is ~/extra/endlesscounting.sh

Lastly, let’s just put it into your personal directory that you created on the home directory, which is

~/[your-directory-name]/

Putting that all together, we get something like:

scp -i ~/extra/GBI-Bioinformatics-Class.pem.sh [email protected]:~/extra/endlesscounting.sh ~/[your-directory-name]/

And there you go! You’ve successfully created a secure connection from your EC2 instance to mine and copied a file from it to your personal directory.

Force Exiting Programs

`(Control + c)`

Let’s now use that file to show how we can force exit programs. The name of the file is endlesscounting.sh, I wonder what it does? Let’s try it out. Navigate to your personal directory and execute the program:

$ cd ~
$ ./endlesscounting.sh

If there is a permission error, simply use the following command (which gives execute permissions for this file to you) and then re-execute the program:

$ chmod +x testprogram.sh

Well.. it appears that the program does exactly what it sounds like, it will just keep counting! Try pressing q to exit!

Ah, no luck. Occasionally this is going to happen to you, or you are going to need to force quit a program because you need to change one of the files or variables you entered into it, or you don’t have enough memory for the whole thing to finish… or maybe you just realized you don’t need to use it.

Hold down the Control button, and then press c! That will exit the program and stop the counting.

The issue to note about this is that if a program is running and in the middle of saving a file or printing something, it will stop the process without caring what has happened. That end result will be your problem! So the program may leave some “loose ends” to follow up on and delete, without telling you what those are.

Virtual Terminals

`tmux`, `screen`

Lastly in this section we are going to create what are called virtual terminals. Remember, a terminal is synonymous with a Command Line Interface console. A “virtual terminal” is a fancy name for a second CLI window on the same computer you are using. The difference between this and just logging into 2 different CLI windows at the same time is that when you open up a virtual terminal, you can log in and out of it without the program you are running in the virtual terminal stopping.

Let’s use the endlesscounting.sh as an example for this. We will pretend that this program that simply counts up endlessly is like an important program that takes a long time. We will create a virtual terminal using tmux, start the program in that new program, and then exit both the virtual terminal and our EC2 instance. Then we’ll SSH back into the EC2 instance, log into our virtual terminal, and see that the program is still running! This means we can launch genome assembly software (or other softwares) and leave them running without us having to be on the computer.

Let’s jump in:

While you can simply use the tmux command alone to create a new virtual terminal, it’s a good idea to name your virtual terminal. The command that we’ll be using to start do this is tmux new -s [name]. This looks like:

$ tmux new -s myterminal

Your CLI will then be taken over by this new virtual terminal. You are still in the same place on your EC2 instance and have access to all the same information. You are simply just looking at it through the window of a new terminal. For example, use ls. You’ll see all the same information in your home directory. Now start the endless counting program:

$ ./endlesscounting.sh

Okay great! But… not that the program is running and we can’t enter text, now how do we exit without force quitting and leaving? I’m glad you asked! In order to leave a virtual terminal, you hold the Command button and then press the letter B on your keyboard, then release those two buttons and press the D button.

Now that you have exited the virtual terminal, let’s also exit the EC2 instance as proof of this principle. Enter:

$ exit

Now we can imagine that we’ve been waiting a while for our program to finish. Lets ssh back into EC2 instance by pressing the up arrow to reload the ssh command and enter it.

Lastly, we want to re-enter the virtual terminal. First, let’s list all the virtual terminals that are running:

$ tmux ls

We can see that our virtual terminal myterminal is still there! Let’s go back into it using:

$ tmux attach-session -t myterminal

The attach-session option tells tmux that you want to reattach to a currently running virtual terminal. The -t option tells tmux that the following word will be the name of the virtual terminal that you want to attach!

Okay we’re back in, we’ve proved that you can run a program behind the scenes without having to watch it run. You can even exit the EC2 instance and turn off your computer and come back later… the program is running in the cloud! Let’s stop endlesscounting.sh with (Control + C).

Review Questions

Which command allows for you to create a secure connection to another computer host (like an EC2 instance)?

To use the Secure Shell Protocol, SSH, you use the command ssh.

What is a private keypair used for?

It acts like a password and is necessary to get permission to use SSH to log into another computer.

Which command would you use to copy a file from your computer to another computer?

To use the Secure Copy Protocol, SCP, you use the command scp.

What information besides the keypair is required to log into another computer system with SSH? What extra information do you need for using SCP?

You need to have the username you will log into the other computer system with and the hostname of the computer. For SCP, you also need to have the path to the location you want to copy your file to or the path and the filename that you want to copy to your own computer.

What happens when you use the command (Control + C) when a program is running?

It force quits the program and send you back to the command prompt!

What command would you use to create a virtual terminal for running programs on behind the scenes on your EC2 instance?

tmux or screen

What do you press to log out of a tmux virtual terminal?

You press and release (Command + B) together and then press D

Which tmux options do you use to reattach to a virtual terminal?

attach-session to tell tmux you want to attach to a currently running virtual terminal and -t [virtual-terminal-name to tell tmux the name of that virtual termianl you want to attach to.

Move on to Section 7: Software Dependencies & More Complex Software Manuals

Go back to tutorial overview