Network Troubleshooting Scenario

Scenario

• Server A and Server B are located within the same subnet.
• Ping connectivity between both servers is successful.
• OCI Security List and NSG rules have already been verified and are correctly configured.
• However, specific ports such as 22 (SSH), 80 (HTTP), 443 (HTTPS), or 8080 (Application Port) are not accessible.
• Since network connectivity and OCI-level security are working as expected, the issue may be related to the Linux firewall (firewalld) running on the target server.

login through bastion-server public Ip address

---

1. Verify Basic Connectivity

The first step is to confirm that network communication between the servers is functioning properly.

From Server A, ping Server B:

ping <Server-B-IP>

Example:

ping 10.0.0.10

Successful output:

64 bytes from 10.0.0.10: icmp_seq=1 ttl=64 time=0.5 ms

now we are pinging from bastion-server to bastion-test-server(using its private ip)

The ping test is successful, indicating that the target host IP is reachable and Layer 3 network connectivity has been established between the source and destination servers.

2. Verify Port Connectivity

After confirming that the target server is reachable via ping, the next step is to verify whether the required application port is accessible from the source server.

Use tools such as Telnet or Netcat (nc) to test connectivity to the target port. This helps determine whether the service is reachable and listening on the expected port.

telnet <Server-B-IP> port

now we are doing telnet from bastion-server to bastion-test-server on port 22

Now, we will perform a Telnet connectivity test from the bastion-server to the bastion-test-server on port 80 to verify network connectivity and confirm that the service is reachable.

3. To check the current status of the firewalld service on the target server, run the following command:

systemctl status firewalld

3. Understanding Systemd Service States (The Core Logic)

When managing services like firewalld using systemctl status firewalld, the combinations of Current State and Future State determine how the server behaves:

• Active / Inactive: Defines the Present (Is the service running right now?).
• Enabled / Disabled: Defines the Future (Will the service start automatically after a reboot?).

Case 1: Enabled and Active (running)

• Current State: The firewall is currently active and protecting the server right now.
• After Reboot: The OS will automatically start the firewall upon booting. No manual intervention is needed.
• Production Environment: This is the standard best practice for live/real working environments to ensure continuous security.

Case 2: Disabled and Active (running)

• Current State: The firewall is currently active because it was started manually (sudo systemctl start firewalld).
• After Reboot: The firewall will not start automatically and will move to an Inactive (dead) state after the server restarts.
• Use Case: Ideal for temporary network troubleshooting or testing environments where you don't want a misconfigured firewall to lock you out permanently if the system reboots.

To enable firewalld

systemctl enable firewalld

To start firewalld

Systemctl start firewalld

To disable firewalld

systemctl disable firewalld

To stop firewalld

systemctl stop firewalld

4.To view the ports currently allowed by the Linux firewall (firewalld), run the following command:

firewall-cmd --list-ports

If the required port does not appear in the list of allowed ports, it has not been permitted through the Linux firewall (firewalld). As a result, traffic to that port may be blocked until an appropriate firewall rule is added.

Understanding firewalld Port Rules

firewalld allows traffic either by port number (for example, 80/tcp) or by service name (for example, ssh, http).

The following command:

firewall-cmd --list-ports

shows only the ports that have been explicitly added using port numbers.

If the output is empty, it simply means that no ports have been added directly by port number. It does not mean that the server is inaccessible, as access may still be allowed through service-based rules.

For example, port 22 does not appear in the --list-ports output, but SSH connectivity still works because the ssh service is enabled in firewalld, which automatically allows traffic on 22/TCP.

5. To allow a specific port through the Linux firewall (firewalld), add a firewall rule for the required port and protocol.

firewall-cmd --permanent --zone=public --add-port=80/tcp

After adding a new port or firewall rule, reload the firewalld configuration to apply the changes without interrupting the running service.

firewall-cmd --reload

Note: A firewall reload is generally preferred over a restart because it applies the updated rules without stopping the firewall service.

Now, verify the firewall configuration again to ensure that the required port has been successfully added.

We can see that the required port is now listed in the firewalld configuration, which confirms that traffic to this port is allowed by the Linux firewall.

6. Now, let's test the port connectivity again to verify that the newly added firewall rule is allowing traffic to reach the target service.

Troubleshooting: Telnet on Port 80 Showing "Connection Refused"

While testing connectivity to Port 80 using the telnet command, the connection was failing with the following error:

telnet <target-private-ip> 80

Output:

Connection refused

Even though Port 80 was already allowed in the OCI Security List(in Ingress and Egress rule) and Linux firewall (firewalld), the connection was still being refused.

Note:- For the Egress Rule, instead of specifying the private IP addresses of the Bastion Test Server, you can use 0.0.0.0/0 as the destination CIDR and select ALL protocols option.

This is generally acceptable because egress rules are typically more permissive than ingress rules, allowing outbound traffic to reach any destination while still maintaining control through ingress restrictions and other security mechanisms.

Root Cause

A "Connection refused" error indicates that the network path is reachable, but there is no application or service listening on the target port.

To verify this, the following command was executed on the target server:

ss -tulnp | grep :80

The command returned no output, confirming that no service was listening on Port 80.

Resolution

An Apache Web Server (httpd) was installed on the target server:

Note: Since the current session is connected to the Bastion Server, running sudo yum install httpd -y here would install Apache on the Bastion Server. To install Apache on the target server, first connect to the Bastion Test Server using its public IP, and then run the command. Once Apache is installed and listening on port 80, Telnet connectivity from the Bastion Server to the Bastion Test Server's private IP on port 80 should succeed.

sudo yum install httpd -y

This command downloads and installs the Apache HTTP Server package.

After installation, the Apache service was started:

sudo systemctl start httpd

Optionally, the service can be enabled to start automatically after a reboot:

sudo systemctl enable httpd

Verification

Verify that the Apache service is running:

systemctl status httpd

Check that Port 80 is now listening:

ss -tulnp | grep :80

Expected output:

LISTEN 0 511 *:80 *:* users:(("httpd",pid=xxxx))

Result

Once the Apache service started listening on Port 80, the Telnet connectivity test succeeded:

telnet <target-private-ip> 80

Key Learning

Allowing a port in the Security List, NSG, or Linux firewall only permits traffic to reach the server. A successful connection also requires an application or service to be actively listening on that port. If no service is listening, the connection will be refused even when all firewall rules are configured correctly.

What Does a Successful Ping Confirm?

• Network connectivity between the servers is working.
• Both servers are reachable within the subnet.
• Routing is functioning correctly.
• OCI Security Lists and NSGs are allowing ICMP traffic.
• Layer 3 (Network Layer) communication is established.

Note: A successful ping only confirms basic network reachability. It does not guarantee that application ports are open or accessible.

Possible Causes of Port Connectivity Issues

Even when ping works successfully, port connectivity may still fail due to:

• OCI Security List restrictions
• NSG restrictions
• Linux firewall (firewalld)
• Application/service not running
• Application not listening on the required port

Linux and Cloud Networking: Firewall Management Guide

When deploying applications in a cloud environment (like Oracle Cloud Infrastructure, AWS, etc.), securing network traffic is handled at two distinct levels: the Cloud Console Level and the OS/Server Level. Understanding the difference between these two layers is critical for a Cloud Architect.

---

1. The Two-Layer Security Model

Think of your Cloud Instance (VM) as a high-security building:

• Cloud Console (Security Lists / NSGs): Acts as the Main Gate of the building (Outside Security).
• OS Level (PuTTY / firewalld): Acts as the Room Door inside the building (Inside Security).

Traffic must pass through both gates to successfully reach an application. If either gate is locked, the connection will fail.

Scenario Analysis from Labs

Scenario A: Port 22 (SSH) works without explicit OS configuration

• Console: Allowed in Ingress Rules (Main Gate Open).
• OS Level (Linux): By default, cloud images come with Port 22 pre-allowed in firewalld so users can connect via PuTTY initially (Room Door Open).
• Result: Connection Successful.

Scenario B: Port 80 (HTTP) fails even after adding to Console and OS

• Console: Allowed in Ingress Rules (Main Gate Open).
• OS Level: Port 80 added to firewalld (Room Door Open).
• Missing Component: No Web Server (like Apache or Nginx) was installed or running to "Listen" on Port 80.
• Result: Connection Refused (No service inside the room).

---

2. Key Differences: Console vs. Server Level

Feature	Cloud Console Level (Security List / NSG)	Server Level (firewall-cmd / PuTTY)
Layer	Cloud Network Level (Outside the Server)	Operating System Level (Inside the Server)
Managed By	Cloud Administrator / Infrastructure Team	System Administrator / Root User
Mechanism	Drops traffic before it hits the VM VNIC	Drops traffic at the OS Kernel level
Default State	Deny All (Strictly Blocked for security)	Essential ports open (e.g., 22), others blocked
Best Use Case	Broad network access control	Granular control (e.g., blocking a specific IP on a single host)

---