When a user reports that “the application is down,” it means the user sent a request but did not receive a response from the application.

When a user sends a request (TCP packet), it goes through several checks before it finally reaches the backend application (Node/Python/Java). The user’s complaint does not reveal where in the chain the failure occurred; it only indicates that the client was unable to complete the request.

---

---

We follow the standard outside-in troubleshooting methodology, starting with Client/Hub Ingress Connectivity. If that fails, we then move to the Private Subnet’s Security List and Routing, followed by the App Server’s OS Firewall, then Verify httpd Running & Listening Status, and finally the Backend Application.

For testing this, there are some prerequisites listed below:

1. Networking setup includes HUB VCN & Public Subnet, SPOKE VCN & Private Subnet, Security Lists, Route Table, Internet Gateway (Hub), Dynamic Routing Gateway, VCN Attachments, and NAT Gateway (Spoke).

   When installing the httpd package on the App Server, we will temporarily need internet access. Therefore, we will create an additional security rule allowing internet traffic through the NAT Gateway, and once the installation is complete, we will remove it.

2. Bastion & Application Server setup

   Now before installing httpd we will update our Security List

   sudo dnf install httpd -y
   

   httpd -v
   

   Our httpd installation is complete, which we have verified using httpd -v. We can now revert to our previous Security Rules.

Testing

1. First, we will test the connection on port 80 of the app_server from the bastion_server using telnet.

   telnet 10.1.0.26 80
   

   A connection hang or Connection timed out indicates that traffic is being dropped at the Private Subnet routing or Security List layer. However, the error we received telnet: connect to address 10.1.0.26: No route to host means our Security List is properly configured, but the OS Firewall of the app_server is blocking the connection.

2. Now we should check on the app_server to determine whether the firewall is active. This can be done using systemctl status firewalld or sudo firewall-cmd --state.

   systemctl status firewalld
   

3. The firewall is active. Now we will check if it is allowing HTTP by using sudo firewall-cmd --list-all.

   sudo firewall-cmd --list-all
   

4. We can see in the above screenshot that the firewall is only allowing SSH and dhcpv6-client. Therefore, we can either deactivate the firewall which is not considered a good practice or explicitly and permanently allow http using sudo firewall-cmd --permanent --add-service=http. After that, we need to reload the firewall to apply the change immediately by running sudo firewall-cmd --reload. Finally, we will verify the change using sudo firewall-cmd --list-all.

   sudo firewall-cmd --permanent --add-service=http
   

   sudo firewall-cmd --reload
   

   sudo firewall-cmd --list-all
   

5. Now we will switch to the bastion_server and repeat step 1 to see if it is working.

   telnet 10.1.0.26 80
   

6. Now it is throwing a new error: telnet: connect to address 10.1.0.26: Connection refused. Therefore, we should check whether httpd is running and listening on port 80. To do this, first switch to the app_server, and then run sudo systemctl status httpd.

   sudo systemctl status httpd
   

7. When we install httpd, it is inactive by default. Therefore, we need to start it using sudo systemctl start httpd and verify its status again. We also need to keep the app_server logged in; if we log out, the Apache HTTP Server might become inactive again.

   sudo systemctl start httpd
   

   sudo systemctl status httpd
   

8. Now it is running, but it should also be listening on port 80 to allow connections. Let’s check this by using: sudo netstat -plan | grep ':80'

   sudo netstat -plan | grep ':80'
   

   We can see that it has started listening as well. By default, when we start the Apache HTTP Server, it also begins listening on port 80.

9. Now we can expect to connect without any errors. First, switch back to the bastion_server or just open a new terminal to log in to bastion_server and let’s try again.

   telnet 10.1.0.26 80
   

   Now we can see its connected finally.

Conclusion