AWS repair, re‐mount instances volume - TheEvergreenStateCollege/upper-division-cs-23-24 GitHub Wiki
If you ever get logged out of your AWS instance by accidentally changing the firewall setting like me or just so happen to lose your PEM files, these are the 2 only ways to save what you already have, based on the experience of someone losing their instances 2 times in a row, yeah that's totally not me...
Mounting volumes (The only way to save if you lost a PEM file)
- Go to your EC2 dashboard (from your instances outside), select
Volumes
, and choose the volume that associates with your instances, in this case, it's my 8gb one
- In this screen, select
Actions
->Detach Volume
-> confirm Detach on the next prompt
- Creates a new instance, after initialization for the example is finished, it's important to go to the
Storage
section under your instance info and take notes on the root device name. When you reattach the volume, the volume's name must be exactly the same as the root device name. If it's not then it'll create some kind of error that the volume can't initialize with the instance
- Detach the volume of the new instance you create. Starting from this process, we'll re-attach our new instances with the old volume. On this same screen where you detach them, choose
Attach Volume
- You will be greeted with this screen. You can choose the new instance you just created. On the device name portion, if they let you choose, choose the root device recommended name (something with
/dev/sda1/
) or if they're not, take note of the root device name on the instance and use that name. Remember, only volumes that are created on a specific region can fit with the instance that stays in the same region as the volume (For example, if the instance is at Ohio US East region then it can only fit with the volume that came from the exact Ohio US East region if it came from Wakanda then yeah the volume must also came from Wakanda too)
- Click on the
Attach volume
button down below, and there you go. Now the volume is already attached to the new instance, just log in to the new instance and try it out!
Assign a session manager (The only way to save if you misconfigured your firewall)
Before I start this, I just want to point out that the firewall can become a dangerous trap if you don't use it correctly. In my experience, I heard some people say that in this case, re-attaching the volume might work, even ChatGPT also somewhat said the same. But it was a false positive. You can't recover this just by switching volumes. It looks like the firewall command (sudo ufw
) has been hardwired to the volume, which makes the volume essentially locked or "bricked" in my definition, the same way as if you have a corrupt drive, that's all. If you follow the instructions for creating an NGINX server around weeks 6-7, they don't explicitly say that you need to open an SSH port when you first use it, which is crucial for logging in. This fix will show you how to remove the firewall temporarily to gain access, however, if you still want to use it, remember to add the port for the SSH by using either sudo ufw allow 22
or sudo ufw allow ssh
- Go to your instance, then go to
Actions
->Security
->Modify IAM roles
- Press
Create new IAM role
- Press
Create role
- Leave everything default but select
EC2
in the use case, then chooseEC2 Role for AWS Systems Manager
right next after you choose EC2
- For the next step, just click
Continue
, leave everything default until you see this screen, give it a name, scroll down, and chooseCreate Role
when you see it
- There, now you have a role that shows up in this portion
- Went back to the previous
Modify IAM roles
screen, select the role that you just created, and clickUpdate IAM roles
. It should now show up on your instance at this line
Now you need to assign yourself as a Host, it's something like setting yourself as an admin so that you can manage your server as a higher-up. To do so, follow these steps
- Go to the System Manager on your AWS. You need to be able to assign yourself as a host to authorize yourself with the instance. Then choose
Create
- Choose 'Create' at
Host Management
, leave everything by default then scroll down and press 'Create'
- The process will take a long time, mostly at the
Collect Inventory
part because it tries to sync with what you have inside your instances. Presumably, it will take around 30 min to an hour and it could be longer but it could also be shorter. The only way to know is that when you refresh the page and the association status shows a full 5/5 green success
- After it shows up, go back to your instances info screen, select 'connect', and check in the
Session Manager
portion, you should have a screen like this. The last thing is to connect by pressingConnect
- When you get in, you'll be greeted with some kind of bash shell like this. Using
pwd
, you will find the location of the terminal at/var/snap/amazon-ssm-agent/7628
or probably any number. Because this is some kind of a "proxy" that connects you to the server in a different way through this agent, you can't access what's inside your volume in general, in my case, that's how I understand it. But you still can executesudo
commands related to changing server infrastructure, including your firewall if you turn it on.
- From here, you can change your firewall, either by including the SSH port as I said before or just disable it by
sudo ufw disable
like in this case. To verify that it works, you can usesudo ufw status
- You're finished. You can't continue to find anything here, even your files, since it's just an agent. The only thing left is to terminate the session and log in again using ssh or whatever other method you have. And there you go, just...remember to take note to open an SSH port every time you enable the firewall, or else your life will become very miserable like the person writing this wiki trying to find a way to do so at 2 am on a random Tuesday