Guide to Troubleshooting Slow Linux Servers

A slow Linux server can have many causes. This guide walks you through a methodical approach to identifying and resolving performance problems, using tools like top, vmstat, and other essential utilities.

The top Command: Your First Diagnostic Tool

The top command is your best ally for starting an investigation. It provides a real-time view of running processes and essential system metrics.

Example top output:

top - 12:41:43 up 52 days, 9:46, 1 user, load average: 0.92, 0.75, 0.79
Tasks: 145 total, 1 running, 144 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5%us, 0.0%sy, 0.0%ni, 99.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4332028k total, 3836264k used, 495764k free, 41952k buffers
Swap: 2097148k total, 1574k used, 2081404k free, 1262632k cached

Step 1: Check I/O Wait and CPU Idle Time

How to Do It

• Use top and look for the “wa” (I/O wait) and “id” (CPU idle) values
• A high I/O wait (>10%) indicates disk access problems
• A low idle time (<25%) suggests CPU overload

Interpretation

• High I/O wait: Potential disk or network problems
• Low CPU idle: Applications consuming too many CPU resources

Step 2: Analyze CPU Usage

If I/O wait is low but idle time is also low:

Identify Resource-Hungry Processes

# Use top and sort by CPU usage
top -o %CPU

# For a specific process
ps -p PID -o %cpu,%mem,cmd

Possible Solutions

1. For abnormal processes:

   # Stop the problematic process
   kill -15 PID
   
   # If it doesn't respond
   kill -9 PID

2. For normal but high usage:

   • Increase server resources
   • Distribute the load across multiple servers

Step 3: Application Problems

When I/O wait is low and CPU idle is high:

Investigation

1. Check the application logs:

   tail -f /var/log/application.log

2. Use strace to analyze system calls:

   strace -p PID

3. Examine network connections:

   lsof -i

Step 4: Memory and Swap Analysis

Check Memory Usage

# Detailed memory usage
free -m

# Real-time monitoring
vmstat 1

Calculating Actually Available Memory

Mémoire Disponible = Libre + Buffers + Cache

Example:

Libre : 495MB
Buffers : 41MB
Cache : 1262MB
Total Disponible = 1798MB

Step 5: Swap Problems

Diagnosis

# View swap usage
free -m

# Detailed monitoring
vmstat 1

Solutions

1. High swap usage:

   • Increase physical RAM
   • Identify processes consuming too much memory

2. Adjust swap settings:

   # Change the tendency to swap
   sysctl vm.swappiness=10

Step 6: I/O Analysis with iotop

Installation and Usage

# Installation
apt install iotop  # Debian/Ubuntu
yum install iotop  # RHEL/CentOS

# Monitoring
iotop -o  # Shows only active processes

Additional Tools

1. vmstat

# Monitoring every second
vmstat 1

# Example output
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0 295388 239748 403468    0    0     0     0  100  100  0  0 100  0

2. Disk Monitoring

# Real-time I/O
iostat -x 1

# Disk space
df -h

Monitoring Best Practices

1. Proactive Monitoring

   • Set up alerts for:
     • CPU usage > 80%
     • Available memory < 20%
     • Swap used > 50%

2. Documentation

   • Keep a history of incidents
   • Note the solutions applied
   • Document normal thresholds

3. Automation

   # Basic monitoring script
   #!/bin/bash
   while true; do
     date >> /var/log/server-metrics.log
     top -bn1 | head -n 10 >> /var/log/server-metrics.log
     sleep 300
   done

Resolving Common Problems

1. High CPU

• Identify resource-hungry processes
• Optimize the code
• Scale vertically or horizontally

2. Insufficient Memory

• Increase RAM
• Optimize applications
• Configure garbage collection

3. Slow I/O

• Check the disks (SMART)
• Optimize database queries
• Use SSD storage

Conclusion

Troubleshooting a slow server requires a methodical approach and the right tools. The key steps are:

1. Identify the specific symptoms
2. Use the appropriate tools
3. Analyze the metrics
4. Apply the fixes
5. Verify the results

Always keep in mind that performance is an ongoing process of optimization and maintenance.

Additional Resources

• Linux Performance Documentation
• Linux Performance Tools Guide
• top Manual

Remember that prevention is often better than cure. Set up proactive monitoring and appropriate alert thresholds.