logo
CHEATSHEETSTOOLSABOUT中文

Linux - System Logs & Journald

Linux logging is split into two primary architectures: the traditional text-based syslog subsystem (rsyslogd storing plain text files in /var/log/) and the modern unified binary logging service systemd-journald (queried via journalctl).

1. Key Log Locations in /var/log

Traditional Unix daemons and system services write text logs directly to /var/log:

Log File Subsystem / Source What It Records
/var/log/syslog (Debian/Ubuntu) or /var/log/messages (RHEL/CentOS) System services & daemons General global system activity, background daemons, non-critical services.
/var/log/auth.log (Debian) or /var/log/secure (RHEL) PAM, SSH, sudo Authentication attempts, SSH logins (successful and failed), privilege escalation.
/var/log/kern.log Linux Kernel Hardware events, driver errors, OOM (Out Of Memory) killer kills, firewall drops.
/var/log/dmesg Kernel boot buffer Boot-time hardware initialization and driver setup messages.
/var/log/audit/audit.log Linux Audit Daemon (auditd) Security events, SELinux denials, seccomp violations, syscall tracking.
/var/log/dpkg.log / apt/ Package managers Package installations, upgrades, and removals.
/var/log/lastlog / /var/log/wtmp Binary login databases Tracks user login sessions (inspected using last and lastlog commands).

2. Modern Binary Logging: systemd-journald

systemd-journald intercepts logs from standard output (stdout/stderr), syslog sockets, kernel ring buffers, and audit streams. Logs are stored as compressed, indexed binary files under /run/log/journal/ (volatile) or /var/log/journal/ (persistent).

Querying Logs with journalctl

journalctl is the command-line utility used to query journal logs with high-performance filtering:

# Follow logs in real-time (equivalent to tail -f)
journalctl -f

# Filter by a specific systemd service unit
journalctl -u nginx.service -f

# Filter by multiple services simultaneously
journalctl -u nginx.service -u php-fpm.service

# Filter by time ranges
journalctl --since "2026-09-30 18:00:00"
journalctl --since "1 hour ago"
journalctl --since yesterday --until "2 hours ago"

# View logs for the current boot only (or previous boot: -b -1)
journalctl -b 0

# Filter by severity level (emerg, alert, crit, err, warning, notice, info, debug)
journalctl -p err..emerg

# Filter kernel logs only (equivalent to dmesg)
journalctl -k

# Output format options (json, short-iso, cat for raw script output)
journalctl -u docker.service -o json-pretty -n 50

3. Kernel Ring Buffer: dmesg

When the Linux kernel boots, file systems are not yet mounted, so messages cannot be written to disk. The kernel writes logs into a circular in-memory buffer called the kernel ring buffer.

# View recent kernel messages
dmesg

# Print human-readable timestamps (-T)
dmesg -T

# Follow kernel logs live (-w)
dmesg -Tw

# Filter by log facility and level (e.g. error, alert)
dmesg -l err,crit

/dev/kmsg provides a character device interface to read from and write to the kernel ring buffer directly from user space.

4. Log Rotation: logrotate

To prevent plain-text log files from filling up disk partitions, Linux uses the logrotate utility, typically executed daily via cron or a systemd timer.

Configuration (/etc/logrotate.d/nginx)

/var/log/nginx/*.log {
    daily                   # Rotate every day
    missingok               # Don't error if file doesn't exist
    rotate 14               # Keep 14 rotated archives before deleting
    compress                # Compress rotated files with gzip
    delaycompress           # Defer compression to next cycle (prevents file locking bugs)
    notifempty              # Don't rotate empty files
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
    endscript
}

The postrotate script sends a signal (SIGUSR1 or SIGHUP) to the process to instruct it to close its old file descriptor and reopen the newly created log file.

Loading...