Essential Linux CLI Tools Every Network Engineer Should Know (Part 1)

Categories: LInux

Introduction

When I transitioned from Windows to Ubuntu for my network engineering work in the data center, one of the biggest revelations was the power of command-line tools. As a data center engineer working extensively with Cisco and Huawei equipment, I spend most of my time SSH'd into remote servers and devices. GUI tools simply aren't practical when you're managing infrastructure remotely.

In this multi-part series, I'll share the essential CLI-based Linux tools that have become indispensable in my daily workflow. Whether you're managing SD-WAN deployments, troubleshooting enterprise networks, or securing your infrastructure, these command-line tools will significantly enhance your capabilities without requiring GUI access.

This is Part 1, where we'll focus on packet analysis and network monitoring tools - all from the command line. If you have basic CCNA knowledge and some familiarity with Linux commands, you're ready to dive in!

Why CLI Tools Matter for Network Engineers

Before we jump into the tools, let me share why CLI-based tools are superior for network engineering:

  • Remote accessibility - Work from anywhere via SSH without X11 forwarding
  • Scriptable automation - Integrate into monitoring scripts and cron jobs
  • Lightweight performance - No GUI overhead consuming resources
  • Universal compatibility - Works on any Linux server, regardless of desktop environment
  • Professional workflow - Matches the CLI-first approach of network devices

Now, let's explore the five essential CLI tools that form the foundation of network analysis on Linux.

 

Tool #1: TShark - Command-Line Packet Analyzer

What it is: TShark is the terminal version of Wireshark, providing powerful packet capture and analysis capabilities entirely from the command line. Think of it as Wireshark without the GUI.

Why you need it: When troubleshooting connectivity issues on remote servers, analyzing application behavior, or detecting security anomalies, TShark provides deep packet-level visibility without requiring GUI access.

Installation

bash

sudo apt update
sudo apt install tshark

During installation, you'll be asked whether non-root users should be able to capture packets. Select "Yes" and add your user to the wireshark group:

bash

sudo usermod -aG wireshark $USER

Log out and log back in for changes to take effect.

if any problem occurs use this commmand:
create the wireshark group automatically is to run the reconfiguration tool:

Bash

sudo dpkg-reconfigure wireshark-common
  1. A purple screen will appear asking: "Should non-superusers be able to capture packets?"

  2. Select Yes and press Enter.

  3. This process will force the system to create the wireshark group.

Basic Usage

Capture packets on a specific interface:

bash

sudo tshark -i ens33

Capture with limited packet count:

bash

sudo tshark -i ens33 -c 100

Save capture to a file:

bash

tshark -i ens33 -w /tmp/capture.pcap

Read from a saved file:

bash

tshark -r /tmp/capture.pcap

Terminal output showing tshark capturing live packets with frame numbers, timestamps, source/destination IPs, protocols, and brief packet info scrolling in real-time.

Essential Display Filters for Network Engineers

TShark uses the same display filter syntax as Wireshark. Here are filters I use daily:

Filter by IP address:

bash

tshark -i ens33 -f "host 192.168.1.100"

Filter by protocol:

bash

tshark -i ens33 -f "tcp"
tshark -i ens33 -f "udp"
tshark -i ens33 -f "icmp"

Filter by port:

bash

tshark -i ens33 -f "tcp port 22"
tshark -i ens33 -f "tcp port 443"

Capture only SYN packets (useful for detecting port scans):

bash

sudo tshark -i ens33 -f "tcp[tcpflags] & tcp-syn != 0"

Terminal showing tshark output with a display filter applied (e.g., "tcp port 22") showing only SSH traffic with detailed packet information.

Advanced TShark Options

Display specific fields only:

bash

tshark -i ens33 -T fields -e ip.src -e ip.dst -e tcp.port

Display packet details in different formats:

bash

tshark -r capture.pcap -V  # Verbose output (like packet details pane)
tshark -r capture.pcap -x  # Include hex dump

Count packets by protocol:

bash

tshark -r capture.pcap -q -z io,phs

Extract HTTP requests:

bash

tshark -r capture.pcap -Y "http.request" -T fields -e http.host -e http.request.uri

Screenshot 3 Needed: TShark verbose output (-V flag) showing detailed packet dissection including Ethernet, IP, and TCP layer information for a single packet.

Real-World Application

When I troubleshoot connectivity issues between SD-WAN devices remotely, I use TShark to:

  • Verify three-way TCP handshakes are completing
  • Check for retransmissions indicating network congestion
  • Identify whether packets are being encrypted properly (by checking port and protocol)
  • Detect unexpected traffic patterns

Pro Tip: Use -Y for display filters (after capture) and -f for capture filters (during capture). Capture filters are faster but less flexible.

Analyzing Captured Traffic

Follow TCP streams from command line:

bash

tshark -r capture.pcap -q -z follow,tcp,ascii,0

Generate statistics:

bash

tshark -r capture.pcap -q -z conv,tcp  # TCP conversations
tshark -r capture.pcap -q -z io,stat,1 # I/O statistics

 

Tool #2: Nmap - Network Mapper and Security Scanner

What it is: Nmap is a powerful network scanning tool used to discover hosts, services, and open ports on a network - all from the command line.

Why you need it: For network inventory, security auditing, and detecting unauthorized services running on your network devices.

Installation

bash

sudo apt update
sudo apt install nmap

Basic Usage

Scan a single host:

bash

nmap 192.168.1.1

Scan a subnet:

bash

nmap 192.168.1.0/24

Scan specific ports:

bash

nmap -p 22,80,443 192.168.1.1

Detect operating system and services:

bash

sudo nmap -O -sV 192.168.1.1

Terminal output of an nmap scan showing open ports on a target device (like a router or switch). Show ports like 22 (SSH), 80 (HTTP), 443 (HTTPS), 179 (BGP) with their states and service versions.

Advanced Scanning Techniques

SYN Stealth Scan (faster and less detectable):

bash

sudo nmap -sS 192.168.1.0/24

Aggressive scan (OS detection, version detection, script scanning):

bash

sudo nmap -A 192.168.1.1

Fast scan (top 100 ports only):

bash

nmap -F 192.168.1.1

Scan all 65535 ports:

bash

nmap -p- 192.168.1.1

Output to different formats:

bash

nmap 192.168.1.1 -oN scan.txt      # Normal output
nmap 192.168.1.1 -oX scan.xml      # XML output
nmap 192.168.1.1 -oG scan.grep     # Greppable output
nmap 192.168.1.1 -oA scan          # All formats

 Nmap scan output with service version detection (-sV flag), showing detailed service information including versions (e.g., OpenSSH 8.2p1, Apache 2.4.41).

Useful Nmap Scripts

Nmap includes NSE (Nmap Scripting Engine) for advanced scanning:

Vulnerability detection:

bash

nmap --script vuln 192.168.1.1

SSH algorithm enumeration:

bash

nmap --script ssh2-enum-algos -p 22 192.168.1.1

SSL/TLS certificate information:

bash

nmap --script ssl-cert -p 443 192.168.1.1

Security Auditing Use Case

I regularly use nmap to:

  • Verify that only authorized ports are open on routers and switches
  • Detect rogue devices on the management network
  • Confirm firewall rules are working correctly
  • Inventory network devices before maintenance windows
  • Check if Telnet (port 23) is disabled on all devices

Tool #3: Netstat - Network Statistics

What it is: Netstat displays network connections, routing tables, interface statistics, and more. It's your window into active network sessions on your Linux system.

Why you need it: To monitor active connections, identify which processes are listening on which ports, and troubleshoot connection issues.

Installation

Netstat is usually pre-installed on most Linux distributions. If not:

bash

sudo apt install net-tools

Essential Commands

Display all active connections:

bash

netstat -a

Show listening ports and associated programs:

bash

sudo netstat -tulpn
  • -t: TCP connections
  • -u: UDP connections
  • -l: Listening sockets
  • -p: Show program name
  • -n: Show numerical addresses (no DNS resolution)

Terminal output of netstat -tulpn command showing listening services with their PIDs, protocols, local addresses, and program names (like sshd, snmpd, systemd-resolved, etc.).

Display routing table:

bash

netstat -r
# or more modern equivalent:
ip route show

Show network interface statistics:

bash

netstat -i

Display all established connections:

bash

netstat -tan | grep ESTABLISHED

Monitoring Active Sessions

See who's logged into your system via SSH:

bash

netstat -tnpa | grep 'ESTABLISHED.*sshd'
# or with more details:
sudo netstat -tnpa | grep ':22.*ESTABLISHED'

Count connections by state:

bash

netstat -ant | awk '{print $6}' | sort | uniq -c | sort -rn
```

This will show output like:
```
    45 ESTABLISHED
    12 TIME_WAIT
     5 LISTEN
     2 CLOSE_WAIT

Screenshot 8 Needed: Output showing established SSH connections with source IPs, ports, and states, demonstrating active remote sessions. Show the command: sudo netstat -tnpa | grep ':22'

Monitor connections to a specific IP:

bash

netstat -an | grep 192.168.1.100

Display connections on a specific port:

bash

netstat -an | grep :443

Advanced Netstat Usage

Continuous monitoring (refresh every 2 seconds):

bash

watch -n 2 'netstat -ant | grep ESTABLISHED | wc -l'

Find which process is using a specific port:

bash

sudo netstat -tulpn | grep :80

Display network statistics:

bash

netstat -s        # All statistics
netstat -st       # TCP statistics only
netstat -su       # UDP statistics only

Output of netstat -s showing network statistics including TCP segments sent/received, retransmissions, failed connection attempts, and reset connections.

Practical Application

In my data center work, I use netstat to:

  • Verify BGP sessions are established (port 179)
  • Monitor HTTPS connections to management interfaces
  • Identify connection states (ESTABLISHED, TIME_WAIT, CLOSE_WAIT)
  • Troubleshoot services that fail to start due to port conflicts
  • Check if SNMP (port 161) is properly listening

Troubleshooting Example:

bash

# Check if SSH daemon is listening
sudo netstat -tulpn | grep :22

# If nothing shows up, SSH isn't running
# Check service status
sudo systemctl status sshd

 

 

 

Tool #4: Tcpdump - Command-Line Packet Capture

What it is: Tcpdump is a powerful packet capture tool that works entirely from the command line. While TShark offers more detailed analysis, tcpdump is faster, lighter, and often pre-installed.

Why you need it: When you need quick packet captures on remote devices, or when you want a lightweight tool for automated monitoring scripts.

Installation

bash

sudo apt update
sudo apt install tcpdump

Basic Usage

Capture packets on an interface:

bash

sudo tcpdump -i ens33

Capture with more readable timestamps:

bash

sudo tcpdump -i ens33 -tttt

Capture and save to a file:

bash

sudo tcpdump -i ens33 -w capture.pcap

Read from a saved file:

bash

tcpdump -r capture.pcap

Terminal showing tcpdump capturing live packets with detailed timestamps (-tttt flag), source/destination IPs, protocols, flags, and packet details.

Powerful Filtering

Capture only SSH traffic:

bash

sudo tcpdump -i ens33 port 22

Capture traffic from a specific host:

bash

sudo tcpdump -i ens33 host 192.168.1.100

Capture traffic between two hosts:

bash

sudo tcpdump -i ens33 host 192.168.1.100 and host 192.168.1.200

Capture only SYN packets (port scan detection):

bash

sudo tcpdump -i ens33 'tcp[tcpflags] & (tcp-syn) != 0 and tcp[tcpflags] & (tcp-ack) == 0'

Capture with more detailed output:

bash

sudo tcpdump -i ens33 -v -n
  • -v: Verbose output (use -vv or -vvv for more verbosity)
  • -n: Don't resolve hostnames (faster)
  • -nn: Don't resolve hostnames or port names

Screenshot 11 Needed: Tcpdump output with verbose flag (-vv) showing detailed packet information including TTL, IP ID, flags, window size, and options.

Advanced Tcpdump Filters

Capture traffic on specific network:

bash

sudo tcpdump -i ens33 net 192.168.1.0/24

Capture traffic NOT from a specific host:

bash

sudo tcpdump -i ens33 not host 192.168.1.1

Capture ICMP (ping) packets:

bash

sudo tcpdump -i ens33 icmp

Capture only TCP RST packets:

bash

sudo tcpdump -i ens33 'tcp[tcpflags] & (tcp-rst) != 0'

Capture packets larger than specified size:

bash

sudo tcpdump -i ens33 greater 1000

Displaying Packet Contents

Show ASCII content:

bash

sudo tcpdump -i ens33 -A port 80

Show both hex and ASCII:

bash

sudo tcpdump -i ens33 -X port 80

Capture specific number of packets:

bash

sudo tcpdump -i ens33 -c 100

 

Practical Tcpdump Scenarios

Monitor DNS queries:

bash

sudo tcpdump -i ens33 -n port 53

Capture only incoming traffic:

bash

sudo tcpdump -i eens33 -Q in

Capture with snaplen (limit captured bytes per packet):

bash

sudo tcpdump -i ens33 -s 96  # Capture only first 96 bytes (headers)

Rotating capture files:

bash

sudo tcpdump -i ens33 -w capture.pcap -C 100 -W 5

This creates files up to 100MB, keeping only the 5 most recent.

Why I Use Tcpdump for Quick Captures

When troubleshooting remote Cisco or Huawei devices, I SSH into a Linux monitoring server and use tcpdump because:

  • It's faster and lighter than TShark for quick captures
  • Often pre-installed on Linux systems
  • Perfect for creating automated capture scripts
  • Can be easily combined with other CLI tools using pipes

Quick troubleshooting workflow:

bash

# Capture 100 packets and analyze
sudo tcpdump -i eth0 -c 100 -w quick.pcap
tcpdump -r quick.pcap | grep SYN | wc -l  # Count SYN packets

Tool #5: Iftop - Real-Time Bandwidth Monitoring

What it is: Iftop displays real-time bandwidth usage on network interfaces in a top-like interface, showing which connections are consuming the most bandwidth - all from the terminal.

Why you need it: To quickly identify bandwidth hogs, monitor interface utilization, and troubleshoot performance issues without GUI tools.

Installation

bash

sudo apt update
sudo apt install iftop

Basic Usage

Monitor default interface:

bash

sudo iftop

Monitor specific interface:

bash

sudo iftop -i ens33

Iftop running in terminal showing active connections with bandwidth usage in real-time. Display should show source/destination hosts, current bandwidth rates (in Kb, Mb), cumulative totals (2s, 10s, 40s averages), and bandwidth bar graphs on the right side.

Understanding Iftop Display

The iftop interface shows:

  • Top section: Source ↔ Destination connections with arrows
  • Right side: Bandwidth usage bars showing relative consumption
  • Three columns: 2 second, 10 second, and 40 second averages
  • Bottom: Total bandwidth statistics (TX, RX, TOTAL)
  • Arrows (=> and <=): Indicate traffic direction

Useful Command-Line Options

Show ports along with IPs:

bash

sudo iftop -i ens33 -P

Don't resolve hostnames (faster):

bash

sudo iftop -i ens33 -n

Don't resolve port names:

bash

sudo iftop -i ens33 -N

Combined (no name resolution):

bash

sudo iftop -i ens33 -nN

Filter by network:

bash

sudo iftop -i ens33 -F 192.168.1.0/24
```

**Screenshot 14 Needed:** Iftop display with port numbers and no name resolution enabled (using `-nNP` flags), showing IP addresses, port numbers, and bandwidth consumption for easier analysis in production environments.

### Keyboard Shortcuts Within Iftop

While iftop is running, use these keys:

- **n**: Toggle hostname resolution on/off
- **s**: Toggle source host display
- **d**: Toggle destination host display
- **p**: Toggle port display
- **P**: Pause display (freeze current view)
- **t**: Toggle display mode (one line, two line, three line)
- **j/k**: Scroll display up/down
- **l**: Display filter prompt
- **q**: Quit

**Using display filter:**
Press `l` and enter a filter like:
```
dst host 192.168.1.1
src host 192.168.1.100
port 443

Screenshot 15 Needed: Iftop with toggled display mode showing detailed breakdown of sent/received traffic separately for each connection pair.

Advanced Iftop Usage

Show cumulative totals only:

bash

sudo iftop -i ens33 -t -s 10
  • -t: Text mode (no ncurses)
  • -s 10: Run for 10 seconds

Output to file for analysis:

bash

sudo iftop -i ens33 -t -s 60 > bandwidth_report.txt

Monitor only incoming or outgoing:

bash

# This is done via filters within iftop using 'l' key
# Example filter: "src host 192.168.1.0/24"

Real-World Monitoring Scenarios

I use iftop daily in these situations:

1. SD-WAN Link Monitoring:

bash

sudo iftop -i ens33 -P

Monitor bandwidth usage during SD-WAN failover testing to ensure traffic shifts properly.

2. Identifying Bandwidth Hogs:

bash

sudo iftop -i ens33 -nNP -o 2s

Quickly identify which IP and port combinations are consuming bandwidth.

3. Verifying Load Balancing: When testing load balancers or link aggregation:

bash

sudo iftop -i bond0 -F 10.0.0.0/8

4. Detecting DDoS or Port Scans: Multiple connections from single IP with small bandwidth each:

bash

sudo iftop -i ens33 -nN

Iftop vs. Other Tools

When to use iftop:

  • Need real-time, interactive bandwidth monitoring
  • Want to see per-connection bandwidth usage
  • Troubleshooting specific bandwidth issues
  • Need visual representation in terminal

When to use alternatives:

  • nethogs: Per-process bandwidth (which program uses bandwidth)
  • nload: Simple interface-level monitoring
  • bmon: More detailed interface statistics
  • vnstat: Long-term bandwidth statistics

Tool Comparison and When to Use Each

Let me share when I reach for each tool in my daily workflow:

Quick Reference Guide

TShark - Use when you need:

  • Deep packet analysis from command line
  • Protocol-level troubleshooting
  • Specific packet field extraction
  • Compatibility with Wireshark filters

Nmap - Use when you need:

  • Network discovery and inventory
  • Port scanning and service detection
  • Security auditing
  • OS fingerprinting

Netstat - Use when you need:

  • Quick check of listening ports
  • View active connections and their states
  • Identify which program uses which port
  • Routing table information

Tcpdump - Use when you need:

  • Fast, lightweight packet capture
  • Quick troubleshooting without analysis overhead
  • Automated capture scripts
  • Minimal system resource usage

Iftop - Use when you need:

  • Real-time bandwidth monitoring
  • Identify bandwidth-hungry connections
  • Visual representation of traffic flow
  • Quick bandwidth anomaly detection

Real-World Use Case: Detecting Unauthorized Access

Let me walk you through a scenario I encountered recently where all these CLI tools worked together to detect and investigate unauthorized access attempts - all done remotely via SSH.

The Scenario

I received an alert about unusual network activity on one of our Linux jump hosts used for network device management. Multiple failed login attempts were logged, and I needed to investigate whether an attacker was attempting to breach our network. I was working remotely and only had SSH access to the server.

Investigation Using Our CLI Tools

Step 1: Check Active Connections with Netstat

First, I SSH'd into the jump host and checked for any established SSH connections:

bash

sudo netstat -tnpa | grep ':22.*ESTABLISHED'
```

**Screenshot 16 Needed:** Terminal output showing netstat command with multiple ESTABLISHED SSH connections, including several from an unusual IP address (e.g., 203.0.113.50) that stands out from normal management IPs.

The output showed:
```
tcp   0   0 10.0.1.50:22        192.168.1.100:54321   ESTABLISHED 1234/sshd: admin
tcp   0   0 10.0.1.50:22        192.168.1.105:54892   ESTABLISHED 1567/sshd: netadmin
tcp   0   0 10.0.1.50:22        203.0.113.50:43891    ESTABLISHED 2891/sshd: [auth]
tcp   0   0 10.0.1.50:22        203.0.113.50:43892    ESTABLISHED 2893/sshd: [auth]
tcp   0   0 10.0.1.50:22        203.0.113.50:43893    ESTABLISHED 2895/sshd: [auth]

I immediately noticed multiple connections from 203.0.113.50 with [auth] status - indicating authentication attempts in progress. This IP wasn't part of our authorized network ranges.

I also checked the connection count:

bash

netstat -ant | awk '{print $6}' | sort | uniq -c | sort -rn

This revealed an unusually high number of TIME_WAIT and SYN_RECV states, suggesting rapid connection cycling - classic brute-force behavior.

Step 2: Monitor Real-Time Activity with Iftop

I launched iftop to see real-time bandwidth patterns:

bash

sudo iftop -i eth0 -nNP

Screenshot 17 Needed: Iftop display showing the suspicious IP (203.0.113.50) with multiple port 22 connections. The bandwidth should be relatively low per connection but showing many concurrent connections - a hallmark of SSH brute-force attempts.

The iftop display showed:

  • Multiple connections from 203.0.113.50 → 10.0.1.50:22
  • Very low bandwidth per connection (few Kb/s)
  • Rapid cycling of connections (appearing and disappearing)
  • Pattern typical of automated scanning/brute-force

This confirmed active attack in progress.

Step 3: Identify the Threat with Nmap

I wanted to profile the attacker's system, so I scanned back:

bash

sudo nmap -A -T4 203.0.113.50

Screenshot 18 Needed: Nmap scan results showing the attacker's open ports and services, possibly revealing it's a compromised Linux server with services like SSH, HTTP, or other indicators of a botnet node.

The scan revealed:

  • Multiple open ports including 22, 80, 445
  • Linux OS (likely a compromised server)
  • SSH service running (suggesting it might also be a victim)
  • No reverse DNS or generic cloud hosting PTR record

Step 4: Capture Attack Traffic with Tcpdump

I started capturing packets from the suspicious IP for evidence:

bash

sudo tcpdump -i eth0 -n host 203.0.113.50 -w /tmp/attack_$(date +%Y%m%d_%H%M%S).pcap

I let this run in a screen session so it would continue capturing even if my SSH session dropped.

After a few minutes, I stopped it with Ctrl+C. The file showed:

bash

ls -lh /tmp/attack_*.pcap
-rw-r--r-- 1 root root 24M Jan 28 14:32 /tmp/attack_20260128_143200.pcap

24MB in just a few minutes - clear indication of high-volume activity.

Step 5: Quick Analysis with Tcpdump

I did some quick analysis of the captured traffic:

bash

# Count total packets
tcpdump -r /tmp/attack_20260128_143200.pcap | wc -l

Result: 18,453 packets in a few minutes!

bash

# Look for SYN packets (connection attempts)
tcpdump -r /tmp/attack_20260128_143200.pcap 'tcp[tcpflags] & (tcp-syn) != 0' | wc -l

Result: 8,234 SYN packets - massive connection attempt volume.

bash

# Check if it's scanning other ports too
tcpdump -r /tmp/attack_20260128_143200.pcap -n | grep -v ':22' | head -20

Screenshot 19 Needed: Terminal showing tcpdump output analyzing the captured pcap file, displaying connection attempts with SYN flags and various destination ports, revealing a port scan pattern.

This revealed the attacker was also scanning ports 23 (Telnet), 3389 (RDP), and others - looking for any vulnerable service.

Step 6: Deep Analysis with TShark

For more detailed protocol analysis, I used TShark:

bash

# Extract statistics about the attack
tshark -r /tmp/attack_20260128_143200.pcap -q -z conv,tcp

This showed multiple short-lived TCP conversations - each attempting to establish SSH connections.

bash

# Look for successful authentications (if any)
tshark -r /tmp/attack_20260128_143200.pcap -Y "ssh.message_code == 52"

Fortunately, no successful authentications were found.

bash

# Analyze timing between connection attempts
tshark -r /tmp/attack_20260128_143200.pcap -Y "tcp.flags.syn==1 && tcp.flags.ack==0" -T fields -e frame.time_relative -e ip.dst -e tcp.dstport | head -20

Screenshot 20 Needed: TShark output showing the timing analysis of SYN packets, demonstrating the rapid-fire nature of the brute-force attack with connections attempted every fraction of a second.

This revealed attempts every 0.1-0.2 seconds - clearly automated.

Step 7: Check Authentication Logs

While not a network tool, I correlated with system logs:

bash

sudo grep "Failed password" /var/log/auth.log | grep "203.0.113.50" | tail -20

This showed hundreds of failed login attempts with common usernames like:

  • root
  • admin
  • administrator
  • user
  • test
  • oracle
  • postgres

Classic dictionary attack pattern.

The Resolution

By combining these CLI tools remotely via SSH, I was able to:

  1. Identify the unauthorized access attempt in real-time (netstat)
  2. Visualize the attack pattern (iftop)
  3. Profile the attacker's system (nmap)
  4. Capture evidence of the attack (tcpdump)
  5. Analyze the attack methodology in detail (tshark)

Immediate Actions Taken

1. Block the attacker immediately:

bash

sudo iptables -A INPUT -s 203.0.113.50 -j DROP

2. Check for any successful logins:

bash

sudo grep "Accepted" /var/log/auth.log | grep "203.0.113.50"

Fortunately, none found.

3. Harden SSH configuration:

bash

sudo nano /etc/ssh/sshd_config

Changes made:

  • Disabled root login: PermitRootLogin no
  • Disabled password authentication: PasswordAuthentication no
  • Limited users: AllowUsers admin netadmin
  • Changed default port: Port 2222

4. Installed fail2ban:

bash

sudo apt install fail2ban
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Configured to ban IPs after 3 failed attempts.

5. Implemented geo-blocking: Since all our management traffic comes from specific countries, I implemented geo-IP blocking:

bash

# Using ipset and iptables to block entire country ranges
# (Implementation details omitted for brevity)

6. Set up monitoring alerts: Created a simple script to alert on suspicious activity:

bash

#!/bin/bash
# /usr/local/bin/ssh_monitor.sh
THRESHOLD=5
COUNT=$(netstat -tn | grep ':22.*SYN_RECV' | wc -l)
if [ $COUNT -gt $THRESHOLD ]; then
    echo "Warning: $COUNT concurrent SSH connection attempts" | mail -s "SSH Alert" admin@example.com
fi

Added to cron:

bash

*/5 * * * * /usr/local/bin/ssh_monitor.sh

Lessons Learned

This incident reinforced several important practices:

  1. CLI tools are sufficient - I handled this entire incident remotely with only SSH access
  2. Layered monitoring - Each tool provided a different perspective
  3. Capture first, analyze later - Starting tcpdump early preserved evidence
  4. Prevention is better - Should have had fail2ban from the start
  5. Documentation matters - Having standard troubleshooting procedures saved time

Creating Your Own Troubleshooting Scripts

One advantage of CLI tools is that you can combine them into powerful scripts. Here are some examples I use:

Network Monitoring Dashboard Script

bash

#!/bin/bash
# network_dashboard.sh - Quick network status overview

echo "=== Network Dashboard ==="
echo

echo "Active Connections:"
netstat -ant | awk '{print $6}' | sort | uniq -c | sort -rn
echo

echo "Top 10 Connections by IP:"
netstat -ant | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -10
echo

echo "Listening Services:"
sudo netstat -tulpn | grep LISTEN
echo

echo "Interface Statistics:"
netstat -i

Screenshot 21 Needed: Output of the network_dashboard.sh script showing the consolidated view of connection states, top connecting IPs, listening services, and interface statistics in one organized display.

Port Scan Detector Script

bash

#!/bin/bash
# detect_scan.sh - Detect potential port scans

INTERFACE="eth0"
THRESHOLD=20  # SYN packets per minute from single IP

echo "Monitoring for port scans on $INTERFACE..."
echo "Threshold: $THRESHOLD SYN packets per minute"

sudo tcpdump -i $INTERFACE -n 'tcp[tcpflags] & (tcp-syn) != 0 and tcp[tcpflags] & (tcp-ack) == 0' 2>/dev/null | \
while read line; do
    IP=$(echo $line | awk '{print $3}' | cut -d. -f1-4)
    echo "$IP $(date +%s)" >> /tmp/scan_detect.tmp
    
    # Check last minute
    CUTOFF=$(($(date +%s) - 60))
    COUNT=$(awk -v ip="$IP" -v cutoff="$CUTOFF" '$1==ip && $2>cutoff' /tmp/scan_detect.tmp | wc -l)
    
    if [ $COUNT -gt $THRESHOLD ]; then
        echo "[ALERT] Possible port scan from $IP ($COUNT SYN packets in last minute)"
        # Optionally block:
        # sudo iptables -A INPUT -s $IP -j DROP
    fi
done

Bandwidth Alert Script

bash

#!/bin/bash
# bandwidth_alert.sh - Alert when bandwidth exceeds threshold

INTERFACE="eth0"
THRESHOLD_MBPS=800
DURATION=10

echo "Monitoring bandwidth on $INTERFACE"
echo "Alert threshold: ${THRESHOLD_MBPS}Mbps"

while true; do
    RESULT=$(iftop -i $INTERFACE -t -s $DURATION -n -N 2>/dev/null | grep "Total send and receive rate" | awk '{print $NF}')
    
    # Convert to Mbps (assuming input is in Mb)
    CURRENT=$(echo $RESULT | sed 's/[^0-9.]//g')
    
    if (( $(echo "$CURRENT > $THRESHOLD_MBPS" | bc -l) )); then
        echo "[ALERT] Bandwidth exceeded: ${CURRENT}Mbps > ${THRESHOLD_MBPS}Mbps"
        # Add notification logic here
    else
        echo "[OK] Current bandwidth: ${CURRENT}Mbps"
    fi
    
    sleep 30
done

Best Practices for CLI-Based Network Troubleshooting

Based on my experience working in data centers with Cisco and Huawei equipment, here are my recommendations:

1. Always Work in Screen or Tmux

When doing network troubleshooting remotely:

bash

# Start a screen session
screen -S troubleshooting

# Or use tmux
tmux new -s troubleshooting

This prevents losing your session if your SSH connection drops.

2. Log Everything

Create a timestamped log directory for each investigation:

bash

mkdir -p ~/investigations/$(date +%Y%m%d_%H%M%S)
cd ~/investigations/$(date +%Y%m%d_%H%M%S)

# Log all commands
script -a commands.log

3. Capture First, Analyze Later

Start packet capture immediately when investigating:

bash

sudo tcpdump -i eth0 -w initial_capture.pcap &
TCPDUMP_PID=$!

# Do your investigation...

# Stop capture when done
sudo kill $TCPDUMP_PID

4. Use Aliases for Common Commands

Add these to your ~/.bashrc:

bash

alias netconns='sudo netstat -tulpn'
alias netactive='sudo netstat -ant | grep ESTABLISHED'
alias netscan='sudo nmap -sS -p-'
alias pktcap='sudo tcpdump -i eth0 -w capture_$(date +%Y%m%d_%H%M%S).pcap'
alias bwmon='sudo iftop -i eth0 -nNP'

5. Document Your Findings

Keep a notes file during investigation:

bash

# investigation_notes.txt
echo "=== Investigation started: $(date) ===" >> notes.txt
echo "Issue: Unusual network activity reported" >> notes.txt
echo "" >> notes.txt

# Add findings as you go
echo "Finding: Multiple connections from 203.0.113.50" >> notes.txt

6. Security Considerations

  • Never capture sensitive data unnecessarily - use capture filters
  • Secure your captures - packet captures may contain sensitive information
  • Limit retention - delete captures after analysis
  • Use encryption - transfer captures over SCP, not unencrypted channels

bash

# Transfer capture securely
scp capture.pcap user@analysis-host:/secure/captures/

# Encrypt sensitive captures
gpg -c sensitive_capture.pcap
shred -u sensitive_capture.pcap  # Securely delete original

Conclusion and Next Steps

These five CLI tools - TShark, nmap, netstat, tcpdump, and iftop - have become absolutely essential in my daily work as a data center engineer. Since transitioning to Ubuntu and embracing the command-line workflow, I've become significantly more efficient at troubleshooting network issues, especially when working remotely.

The beauty of these CLI tools is that they:

  • Work everywhere - Any Linux server with SSH access
  • Are scriptable - Integrate into automation workflows
  • Require minimal resources - No GUI overhead
  • Provide deep insights - Direct access to network data
  • Are battle-tested - Used by network engineers worldwide

What's Coming in Part 2

In the next installment of this series, we'll explore more advanced CLI tools:

  • mtr & traceroute - Advanced path analysis and latency troubleshooting
  • ss - The modern replacement for netstat with more powerful filtering
  • ip command suite - Modern network configuration and management
  • ethtool - Interface-level diagnostics and tuning
  • iperf3 - Network performance testing
  • nethogs - Per-process bandwidth monitoring

Practice Exercise for This Week

To get comfortable with these tools, try this hands-on exercise:

Day 1-2: Setup and Basic Usage

  1. Install all five tools on your Ubuntu system
  2. Practice basic commands for each tool
  3. Create the aliases mentioned in best practices

Day 3-4: Monitoring Your Own Network

  1. Use iftop to monitor your network for 1 hour
  2. Run nmap scan against your own devices (with permission!)
  3. Use netstat to identify all listening services
  4. Capture 10 minutes of traffic with tcpdump

Day 5-6: Analysis Practice

  1. Open your tcpdump capture with TShark
  2. Practice using different display filters
  3. Extract specific protocol statistics
  4. Create a simple report of your findings

Day 7: Integration

  1. Combine multiple tools to investigate a specific scenario
  2. Create one of the monitoring scripts mentioned above
  3. Set up automated monitoring on a test system

Resources for Further Learning

Man Pages - Your best friend:

bash

man tshark
man nmap
man netstat
man tcpdump
man iftop

Online Resources:

Practice Environments:

  • Set up lab environments using VirtualBox or VMware
  • Practice in your own PNETLab or GNS3 setup
  • Use controlled test environments before production

Final Thoughts

The transition from GUI tools to CLI-based troubleshooting might seem daunting initially, but the investment pays off tremendously. These tools have made me a more effective network engineer, whether I'm troubleshooting SD-WAN tunnels, investigating security incidents, or optimizing network performance.

The real power comes from understanding how to combine these tools. Each tool provides a different lens through which to view network behavior, and using them together gives you complete visibility into your network infrastructure.

Whether you're managing Cisco routers, Huawei switches, or Linux servers, these CLI tools will serve you well throughout your career. Master them, and you'll be equipped to handle virtually any network troubleshooting scenario from anywhere with just an SSH connection.

Stay tuned for Part 2 where we'll dive deeper into advanced CLI tools for network engineers!


About the Author: I'm a data center engineer specializing in SD-WAN deployments and working extensively with Cisco and Huawei networking equipment. I transitioned from Windows to Ubuntu for my daily work and have embraced CLI-based workflows. Follow my technical journey and insights at tawkirjournal.com.