Complete Guide to Server Troubleshooting Commands

Detailed guide to the essential commands for server troubleshooting, with a focus on logging.

Basic Logging Commands

1. Real-Time Log Monitoring

# Follow system logs
tail -f /var/log/syslog

# Follow authentication logs
tail -f /var/log/auth.log

# Follow Apache logs
tail -f /var/log/apache2/access.log
tail -f /var/log/apache2/error.log

# Follow Nginx logs
tail -f /var/log/nginx/access.log
tail -f /var/log/nginx/error.log

# Follow MySQL logs
tail -f /var/log/mysql/error.log

2. Log Filtering

# Filter by error
grep "ERROR" /var/log/syslog

# Filter by date
grep "$(date +%Y-%m-%d)" /var/log/syslog

# Filter with context
grep -A 5 -B 5 "ERROR" /var/log/syslog

# Filter with exclusion
grep -v "INFO" /var/log/syslog

# Filter with count
grep -c "ERROR" /var/log/syslog

3. Log Analysis

# Display the last lines
tail -n 100 /var/log/syslog

# Display the first lines
head -n 100 /var/log/syslog

# Display with line numbers
cat -n /var/log/syslog

# Display with highlighting
grep --color=auto "ERROR" /var/log/syslog

# Display with timestamps
awk '{print $1, $2, $3}' /var/log/syslog

Advanced Logging Commands

1. Log Analysis with Awk

# Count errors per hour
awk '{print $3}' /var/log/syslog | grep "ERROR" | sort | uniq -c

# Extract IPs from Apache logs
awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c

# Analyze HTTP status codes
awk '{print $9}' /var/log/apache2/access.log | sort | uniq -c

# Average response time
awk '{sum+=$10} END {print sum/NR}' /var/log/apache2/access.log

2. Log Analysis with Sed

# Remove empty lines
sed '/^$/d' /var/log/syslog

# Replace text
sed 's/ERROR/ERREUR/g' /var/log/syslog

# Extract specific lines
sed -n '100,200p' /var/log/syslog

# Remove lines containing a word
sed '/ERROR/d' /var/log/syslog

3. Log Analysis with Perl

# Extract errors with context
perl -ne 'print if /ERROR/ .. /^$/;' /var/log/syslog

# Analyze logs with regex
perl -ne 'print if /(ERROR|WARNING)/' /var/log/syslog

# Format logs
perl -pe 's/(\d{4}-\d{2}-\d{2})/$1/g' /var/log/syslog

System Monitoring Commands

1. Resource Monitoring

# CPU monitoring
top
htop
mpstat 1

# Memory monitoring
free -m
vmstat 1

# Disk monitoring
df -h
iostat 1

# Network monitoring
netstat -tulpn
iftop

2. Process Monitoring

# List processes
ps aux

# Process tree
pstree

# Process details
lsof -i
lsof -p <PID>

# Process priority
nice -n -20 <command>

3. Service Monitoring

# Service status
systemctl status <service>

# Service logs
journalctl -u <service>

# Follow logs in real time
journalctl -u <service> -f

# Filter logs by priority
journalctl -p err

Diagnostic Commands

1. Network Diagnostics

# Connectivity test
ping <host>
traceroute <host>

# Port test
nc -zv <host> <port>
telnet <host> <port>

# Network analysis
tcpdump -i eth0
wireshark

2. System Diagnostics

# System information
uname -a
lsb_release -a

# Hardware information
lshw
dmidecode

# Network information
ifconfig
ip addr

3. Performance Diagnostics

# System profiling
strace <command>
ltrace <command>

# Performance analysis
perf top
perf record

# Memory analysis
valgrind <program>

Best Practices

1. Log Organization

   • Rotate logs regularly
   • Compress old logs
   • Archive important logs

2. Log Analysis

   • Set up filters
   • Create reports
   • Monitor proactively

3. Maintenance

   • Clean up regularly
   • Check permissions
   • Update configurations

Useful Scripts

1. Monitoring Script

#!/bin/bash
# Real-time log monitoring
tail -f /var/log/syslog | while read line; do
    if echo "$line" | grep -q "ERROR"; then
        echo "[$(date)] $line" >> /var/log/errors.log
    fi
done

2. Analysis Script

#!/bin/bash
# Apache log analysis
awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c | sort -nr | head -n 10

3. Maintenance Script

#!/bin/bash
# Log rotation
find /var/log -type f -name "*.log" -mtime +7 -exec gzip {} \;

Conclusion

Server troubleshooting requires a solid command of logging and analysis commands. This guide covers the essential commands and advanced techniques for effective log management.

Additional Resources

• Documentation Systemd Journal
• Guide Linux Logging
• Tutorial Log Analysis
• Linux System Monitoring
• Log Management Best Practices