/var/log (System Log Directory)

# -----------------------------------------------
#  Real-time log monitoring
# -----------------------------------------------

# tail -f {log-file}
#   → Displays lines appended to the end of the file in real time
#   → Press Ctrl+C to stop
#   Example: sudo tail -f /var/log/syslog

# tail -n {lines} {log-file}
#   → Displays the last N lines of the file
#   Example: sudo tail -n 100 /var/log/auth.log

# -----------------------------------------------
#  Keyword search in logs
# -----------------------------------------------

# grep {pattern} {log-file}
#   → Extracts lines matching a keyword from a log file
#   Example: sudo grep "Failed password" /var/log/auth.log

# grep -i {pattern} {log-file}
#   → Searches case-insensitively
#   Example: sudo grep -i "error" /var/log/syslog

# grep -E "{pattern1}|{pattern2}" {log-file}
#   → Searches for multiple keywords using OR logic
#   Example: sudo grep -E "error|warn|crit" /var/log/syslog

# -----------------------------------------------
#  Searching rotated logs
# -----------------------------------------------

# zcat {.gz file}
#   → Decompresses and displays a gzip-compressed rotated log
#   Example: sudo zcat /var/log/syslog.2.gz

# zgrep {pattern} {.gz file}
#   → Searches a compressed log file directly without decompressing it
#   Example: sudo zgrep "Failed password" /var/log/auth.log.2.gz

# -----------------------------------------------
#  Viewing logs with a pager
# -----------------------------------------------

# less {log-file}
#   → Opens a log file in the pager
#   → Press Shift+F to switch to real-time follow mode (equivalent to tail -f)
#   → Type / followed by a keyword to search
#   → Press q to quit
#   Example: sudo less /var/log/nginx/error.log

# -----------------------------------------------
#  Integration with the systemd journal
# -----------------------------------------------

# journalctl -u {service-name} -f
#   → Streams logs for a systemd-managed service in real time
#   → These logs are stored in binary format, not as files under /var/log
#   Example: sudo journalctl -u nginx -f
#   See the journalctl page for details

File / Directory	Description
/var/log/syslog	General-purpose system log. Records messages from the kernel and various daemons. Used on Debian/Ubuntu-based systems.
/var/log/messages	System-wide log equivalent to syslog. Used on RHEL/AlmaLinux/CentOS-based systems.
/var/log/auth.log	Records all authentication-related events, including SSH logins, sudo usage, and PAM authentication. Used on Debian/Ubuntu-based systems.
/var/log/secure	Authentication log equivalent to auth.log. Used on RHEL/AlmaLinux/CentOS-based systems.
/var/log/kern.log	Records messages output by the kernel. Useful for investigating hardware errors and driver warnings.
/var/log/dmesg	Stores kernel messages output during boot. The same information can also be viewed with the dmesg command.
/var/log/nginx/	Nginx log directory. Contains access.log (request records) and error.log (error records).
/var/log/nginx/access.log	Records every HTTP request made to Nginx. Includes IP address, request details, response code, and byte count.
/var/log/nginx/error.log	Records error messages output by Nginx. Useful for diagnosing configuration errors, PHP-FPM connection failures, and permission issues.
/var/log/apache2/	Apache log directory. Contains access.log and error.log.
/var/log/mysql/	MySQL/MariaDB log directory. Contains the error log and slow query log.
/var/log/mail.log	Log for mail sending and receiving. Records messages from mail transfer agents such as Postfix and Sendmail.
/var/log/cron	Execution records for cron jobs. Lets you verify that scheduled tasks are running correctly.
/var/log/dpkg.log	Records the history of package installations, removals, and updates via apt/dpkg. Used on Debian/Ubuntu-based systems.
/var/log/lastlog	Binary file recording the last login time for each user. Read its contents with the lastlog command.
/var/log/wtmp	Binary file storing the history of logins and logouts. Read its contents with the last command.
/var/log/btmp	Binary file storing failed login attempts. Read its contents with the lastb command.

Investigate unauthorized SSH login attempts

# -----------------------------------------------
#  Investigate unauthorized login attempts on the server
# -----------------------------------------------

# Extract failed password authentication attempts from auth.log
# (Use /var/log/secure on RHEL-based systems)
sudo grep "Failed password" /var/log/auth.log

# Count attempts per source IP address
# awk extracts the IP, then sort | uniq -c tallies the results
sudo grep "Failed password" /var/log/auth.log \
  | awk '{print $(NF-3)}' \
  | sort | uniq -c | sort -rn | head -10

# Filter for attempts targeting only user shinji
sudo grep "Failed password for shinji" /var/log/auth.log

Run the following command:

$ sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -5
    482 203.0.113.42
    317 198.51.100.7
    201 192.0.2.88
     94 203.0.113.101
     12 192.0.2.5

Filter errors from the Nginx error log

# -----------------------------------------------
#  Investigate Nginx errors on the site
# -----------------------------------------------

# Check the last 50 lines of error.log
sudo tail -n 50 /var/log/nginx/error.log

# Extract only messages at [error] level or above
sudo grep "\[error\]" /var/log/nginx/error.log

# Extract only PHP-FPM connection failures (connect() failed)
sudo grep "connect() failed" /var/log/nginx/error.log

# Find HTTP 5xx error requests in the access log
# Extracts lines where the response code starts with 5
sudo awk '$9 ~ /^5/' /var/log/nginx/access.log | tail -20

Run the following command:

$ sudo grep "connect() failed" /var/log/nginx/error.log
2026/03/25 14:32:01 [error] 1234#1234: *5678 connect() failed (111: Connection refused) while connecting to upstream, client: 203.0.113.10, server: rei.example.com, request: "GET /api/status HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000"

Search past logs including rotated files

# -----------------------------------------------
#  Search logs from the past week on the server
# -----------------------------------------------

# /var/log/auth.log* includes rotated historical logs as well
# Search both the current file and compressed archives at once
sudo grep "Accepted publickey for asuka" /var/log/auth.log*

# Search a compressed file with zgrep (no need to decompress the .gz file)
sudo zgrep "Accepted publickey for asuka" /var/log/auth.log.2.gz

# List all rotated log files
ls -lh /var/log/auth.log*

Run the following command:

$ ls -lh /var/log/auth.log*
-rw-r----- 1 syslog adm  48K Mar 25 23:59 /var/log/auth.log
-rw-r----- 1 syslog adm  62K Mar 18 23:59 /var/log/auth.log.1
-rw-r----- 1 syslog adm  18K Mar 11 23:59 /var/log/auth.log.2.gz
-rw-r----- 1 syslog adm  14K Mar  4 23:59 /var/log/auth.log.3.gz

Check sudo usage history

# -----------------------------------------------
#  Review sudo usage records for a user
# -----------------------------------------------

# Extract sudo usage logs from auth.log
sudo grep "sudo:" /var/log/auth.log

# Show only sudo usage records for user misato
sudo grep "sudo:.*misato" /var/log/auth.log

# Extract sudo authentication failures (wrong password) for misato
sudo grep "authentication failure.*misato" /var/log/auth.log

Run the following command:

$ sudo grep "sudo:.*misato" /var/log/auth.log
Mar 25 09:14:02 nerv-srv sudo: misato : TTY=pts/1 ; PWD=/home/misato ; USER=root ; COMMAND=/usr/bin/systemctl restart nginx
Mar 25 11:40:17 nerv-srv sudo: misato : TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/tail -f /var/log/nginx/error.log

Check cron job execution records

# -----------------------------------------------
#  Verify that scheduled batch jobs are running correctly
# -----------------------------------------------

# Check the cron execution log
# Debian/Ubuntu: cron entries are mixed into /var/log/syslog
sudo grep "CRON" /var/log/syslog

# RHEL-based systems: a dedicated file at /var/log/cron
sudo tail -n 50 /var/log/cron

# Extract jobs that did not produce a CMD entry (possible failures)
sudo grep "CRON" /var/log/syslog | grep -v "CMD"

Run the following command:

$ sudo grep "CRON" /var/log/syslog | tail -5
Mar 25 03:00:01 nerv-srv CRON[4501]: (kaji) CMD (/home/kaji/bin/backup.sh >> /home/kaji/logs/backup.log 2>&1)
Mar 25 04:00:01 nerv-srv CRON[4812]: (kaji) CMD (/home/kaji/bin/report.sh)
Mar 25 05:00:01 nerv-srv CRON[5103]: (kaji) CMD (/home/kaji/bin/backup.sh >> /home/kaji/logs/backup.log 2>&1)
Mar 25 06:00:01 nerv-srv CRON[5441]: (kaji) CMD (/home/kaji/bin/report.sh)
Mar 25 07:00:01 nerv-srv CRON[5782]: (kaji) CMD (/home/kaji/bin/backup.sh >> /home/kaji/logs/backup.log 2>&1)

The log files under /var/log are the starting point for understanding the state of a Linux server, investigating incidents, and conducting security audits. Log files are rotated periodically by logrotate (compressed and managed by generation) to prevent disk space from being exhausted. The logrotate configuration is stored in /etc/logrotate.conf and /etc/logrotate.d/. On systems using systemd, logs for many services are managed in a binary journal format rather than as plain files. In that case, use the journalctl command to access them. Log forwarding and aggregation is handled by rsyslog (/etc/rsyslog.conf), which also supports centralizing logs on a remote server. To automatically block unauthorized SSH access, combine fail2ban with auth.log so that offending IP addresses are blocked automatically. For details on password authentication and public key authentication, see the passwd / shadow and ssh / authorized_keys pages.