Basics of Network Connectivity Troubleshooting

“Why didn’t the packet cross the road?”

Introduction

In this guide, we will explore network connectivity troubleshooting using a simple use case. While the focus is primarily on Linux troubleshooting, we will also incorporate relevant Windows tools when applicable.

Please note that this guide is not exhaustive and should be considered a starting point for troubleshooting. The objective is to provide you with tools and techniques that can serve as a foundation for your troubleshooting efforts.

Let’s begin by taking a look at the network diagram.

Our Network

Below is a diagram illustrating a simple network that we can reference throughout this guide. It is important to remember that the components between your source and destination hosts may not always be known. We will discuss techniques to help identify potential issues with your source host, destination host, or anything in between.

In this example network, we have a source workstation, a destination server (bearlychilly.com), and a firewall in between. While we assume both the source and destination hosts are Linux-based, we will also highlight relevant Windows utilities during the troubleshooting steps.

The Scenario

Here is the scenario we will address:

The destination server has a service listening on port 514/TCP to receive logs from other hosts. Our source workstation is configured to send logs to the destination server, but the server’s service is not receiving the logs.

Our Approach

To address this issue effectively, we will break it down into several smaller questions that we can answer. Here are some sample questions that may arise:

• Can we establish a connection from the workstation to the server?
• Is the server listening on port 514/TCP?
• Is the traffic reaching the server?
• Is the firewall between the two components affecting the traffic unexpectedly?

To answer these questions, we will utilize various tools. Here are the utilities and tools we will employ:

• ping (Linux/Windows) - Checks network connectivity
• nslookup (Linux/Windows) - Checks for DNS issues
• netstat (Linux/Windows) - Verifies listening services and ports
• WireShark (Linux/Windows) - Monitors network traffic
• netcat (Linux) - Creates listeners and sends network traffic
• tcpdump (Linux) - Monitors network traffic
• test-netconnection (Windows) - Checks TCP and Ping connectivity

Our Actions

Can We Reach It?

Question: Can we establish a connection from the workstation to the server?

The simplest way to check network connectivity is by using the ping command. To perform a connectivity check using ping, open a terminal or command prompt and execute the following command, replacing bearlychilly.com with the host you want to ping:

ping bearlychilly.com

When pinging the host name, DNS resolution for the requested host is also attempted. If the ping fails, it may indicate that your DNS server cannot resolve the host name. In such cases, ensure that there is a DNS entry for the server’s hostname and verify that the workstation is using the correct DNS server. You can test DNS resolution alone using the nslookup command. The following examples demonstrate DNS resolution, with the first command using the default server and the second allowing you to specify the DNS server (1.1.1.1 in this example):

nslookup bearlychilly.com
nslookup bearlychilly.com 1.1.1.1

If the ping fails, it could indicate that the workstation is unable to reach the server. Keep in mind that some networks block ICMP traffic, which may cause the ping to fail. Despite the initial ping test result, it is recommended to continue troubleshooting.

For our example use case, let’s assume that the ping was successful and the DNS resolution returned the correct IP address for the host name.

Is It Listening?

Question: Is the server listening on port 514/TCP?

Since the ping was successful, we now need to verify whether the server is actively listening on TCP port 514. To do this, we can use the following netstat commands:

Linux

netstat -an | grep LISTEN

Windows PowerShell

netstat -an | Select-String -Pattern "LISTEN"

After executing this command, you should see a list of ports and associated protocols that are actively listening. Confirm that TCP port 514 is included in the list to ensure it is ready to receive traffic. If you do not find port 514 TCP, contact the application administrator and verify if the application is correctly configured to listen on that port and protocol.

As observed in the screenshot above, TCP port 514 is actively listening. We have successfully pinged the host earlier. Therefore, there must be another issue between the data sent from the workstation and the application on the server.

Next, let’s investigate whether the TCP traffic is reaching the server.

Is It Reaching?

Generating Traffic

Question: Is the traffic reaching the server?

Various tools can monitor traffic on the server, but the most common options are tcpdump for Linux and Wireshark for Windows.

Note: TCPdump is capable of monitoring both TCP and UDP traffic.

Let’s assume that the workstation’s traffic occurs infrequently. In such cases, we need to generate traffic ourselves.

Linux On Linux, the netcat utility (or nc as an alias) can be used to send TCP and UDP traffic. When using the -v flag for TCP, you will receive verbose output indicating a successful connection establishment. For UDP, you will not see any output due to the nature of UDP. In both scenarios, you will be greeted by a blank like for text, which will be sent to the specified destination and port.

# TCP
nc bearlychilly.com 514 -v

# UDP
nc -u bearlychilly.com 514

Windows On Windows, the Test-NetConnection cmdlet is super useful and has many different utilities built-in. The command below initiates a ping test and then attempts a TCP handshake with the specified host and port.

Test-NetConnection -ComputerName bearlychilly.com -Port 514

Now that we have a method for generating traffic, let’s proceed to the server and start monitoring. If your source is sending data frequently, there is no need to generate your own traffic.

Monitoring Traffic

Let’s set up the server to monitor incoming traffic.

Linux On Linux, we can utilize tcpdump to monitor network traffic. By using the -i any option, tcpdump captures packets from any available network interface. The filter dst port 514 specifically targets packets with a destination port of 514. You can further enhance the filter by specifying the src if needed to narrow down the traffic.

tcpdump -i any dst port 514 and src <source_ip>

For more specific options, refer to Daniel Miessler’s guide on tcpdump.

Windows Wireshark is the recommended network monitoring tool for Windows. Although it may seem complex initially, it is relatively easy to learn, especially for beginners. Proficiency in monitoring traffic and analyzing pcap files using Wireshark is a valuable skill. Consider exploring additional guides or video tutorials to familiarize yourself with Wireshark’s usage.

Firewall Issues?

Question: Is the firewall between the two components affecting the traffic unexpectedly?

With network monitoring set up, you can revisit the previous section and generate traffic using nc or Test-NetConnection if necessary. It is important to note that if your source is already sending data frequently, you may already observe traffic. Additionally, there are numerous other methods to generate traffic; the provided examples are good starting points.

Now that we can monitor traffic on the server, you should be able to observe TCP flags associated with the network traffic. I recommend doing some research into the TCP handshake and TCP flags if you are not familiar with the topic.

Possibility 1 If you observe TCP flags indicating connection resets, it may imply that you either lack host-based firewall rules allowing traffic into the server or that the application itself is terminating the connection.

Possibility 2 If, after verifying the correctness of the entered commands, you do not observe any traffic, it is likely that a component between the workstation and the server is blocking the traffic. For this use case I would need to configure the firewall between the workstation and the server to allow the traffic from the source to the destination over TCP port 514.

Conclusion

This guide has addressed a relatively simple use case to demonstrate how various tools can be leveraged for network connectivity troubleshooting. I hope this serves as a good starting point to build your toolkit for performing similar work.