First Things First

I am going to assume that you have already set up your VPS. My personal recommendation is Digital Ocean, but I like Linode as well. You can go ahead and attach your Domain Name (if you have one) to the IP you are handed out if you'd like.
Another assumption is that you are connecting to these hosts through a Windows device. We will use this device to generate our SSH keys, get them to the remote host, and I will show you a few other tricks for SSH in Windows.

You can connect to your host by using the following in a terminal
ssh root@yourVPSIP
so for example: ssh [email protected]
You should be prompted to enter any password you set. Since this guide will cover setting up SSH keys I am going to assume you are not using them yet.

Once you are in, do the thing you should always do when connecting to a new system and update!
sudo apt update && sudo apt upgrade -y

Setting Up a New User

Initially root will be the only user account on the system, but we will go ahead and configure a new user to fall in line with some security best practices. Let's create a new user, give them a home directory, add them to the necessary groups, and set their default shell to bash by using this command
useradd -m -G sudo,admin -s /bin/bash yourusernamehere

Now we need to give this user a password. Think of a secure password (14 or more characters, including capital letters, numbers, and special characters) and use the following command
passwd yourusernamehere
After this you should be able to swap to the new user using
su yourusernamehere
If you see yourusernamehere@yourhost:~$ or something similar then you have followed these steps correctly. We will move on to configuring SSH, which will not be entirely on your remote host.

Proceed to: Configuring SSH