Resolving SSH issues through serial console connection

If we are unable to login to an instance if SSH has issues like (Public/Private) keys are lost, password authentication is not set or SSHD config file have some issue.

There are 2 solutions to address the instance login issue:

If the problematic instance (Instance 1) is in the same subnet (part of same VCN) as another healthy instance (Instance 2), then we can use the healthy instance to reach the problematic instance.

If problematic instance is not connected to any other instance, we need to use the Cloud Shell feature.

Note: The recommended solution is to use the Cloud Shell feature.

Note: If both instances are in the same subnet, we can use healthy Instance to access problematic Instance.

Access the problematic instance using a healthy instance.

11_1: Identify unhealthy and healthy instance.

We have 2 test instances here. Instance1 (test 9) is unhealthy and Instance2 (test 8) is healthy instance.

11_2: Login to healthy instance through putty.

Here Test8 is a healthy instance, we will login through putty and try to recover access to unhealthy instance from this.

11_3: Generating ssh key.

We will go to ssh folder and look for any key saved in the authorized keys in .ssh folder.

Check the loaction.

pwd

List all the files and folders.

ls -ltrha

Go to .ssh folder.

cd .ssh

List again files and folders.

ls -ltrh

Read the authorized key folder.

cat authorized_keys

If not present, then we will create public and private key using command ssh-keygen.

ssh-keygen

ls -ltrh

Now we can see 2 files- public and private key are generated. id_rsa.pub is the public key. id_rsa is the private key.

After that we will copy the public key using cat command.

cat id_rsa.pub

11_4: Creating local connection

And then, open the Problematic instance (Test9) in the console.

Scroll down, select Console Connection, and then choose the Create Local Connection option

Now, choose the Paste Public Key option and paste the public key that we generated in the Healthy instance(Test8) using the ssh-keygen command.

Console connection is created and active.

Click on three dot in the right corner of created console connection and select copy serial console connectuon for linux/mac

Execute this copied command in putty.

(Note: Before executing this command, ensure it is executed where the public and private keys were created i.e healthy instance)

Now go to the problematic instance and Force Reboot in console

And then immediately switch to putty and keep pressing esc (escape key) till we get the below screen. Use arrow key to reach Boot Manager menu and hit enter

Choose Oracle Linux and hit enter

Choose Oracle linux version 8. Press enter And then immediately press esc

Press e for edit.

Use right arrow to reach end of line And press space bar and then add init=/bin/bash Press ctrl+x to start.

Now we reached to the root of problematic instance(Instance9)

As we can see, now we are in local host of problematic instance(Instance9) from there we can fix the issue in sshd file

But, as we can see in below screen we have Read only permission for sshd_config file

so, first we have to change the permission using /bin/mount -o remount, rw / command

/bin/mount -o remount, rw /

Then, run the copy command to overwrite the original file (sshd_config) with the backup file (sshd_config_bkp)

cp sshd_config_bkp sshd_config

Compare both files by diff command and there are no differences.

diff sshd_config sshd_config_bkp

Now, switch to the console and delete the created local connection

Then Force reboot the problematic instance (Test9)

We can see in putty that we are now out of local host of problematic instance (Test9) and automatically switched to healthy instance (Test8)

Finally, log in to problematic instance with OPC user and reset the password log in for the same

For changing password, please refer to https://github.com/Nirvan-Pandey/OCI_DOC/wiki/7_Enabling_Password_Based_Access_For_User_Root