Analyzing System Log

systemd has a centralized logging mechanism that collects logs from all user processes in the system and kernel. It is called journal. systemd runs a journal daemon journald, which collects messages from the kernel, initrd (initial RAM disk), services, and etc.

Analyzing logs is the most effective way to diagnose issues with any systemd service. Ghaf's default systemd log level is set to info. To gain deeper insights into the service state, the log level can be elevated to debug with the following option:

ghaf.systemd.logLevel = "debug";

While it is possible to elevate the log level on a live system using systemctl, this option is particularly useful when you need to inspect the startup sequence of critical services that cannot be restarted in a live environment.

To change the log level to debug, run the following systemctl command:

$> sudo systemctl log-level debug

It will change the log level for the systemd daemon and all systemd managed services.

After adjusting the log level, we recommended to reload the systemd daemon and restart the service you are debugging.

Using journalctl

When journalctl command is run without any option, it will show all the messages, which can be pretty long.

  • You can see logs of specific boot using -b option for example:

    $> journalctl -b  #Log from current boot
$> journalctl -b -1 #Log from previous boo

  • To list available boots, use the following command:

    $> journalctl --list-boots

  • To view the logs generated by any systemd unit, use the -u option. For example, the command below displays all logs recorded by the logind service. You can specify multiple units by using the -u switch more than once.

    $> journalctl -u logind.service

  • You can see log messages in real-time, similar to the tail command in Linux. To do this, use the -f option:

    $> journalctl -f

  • Similar to the tail command, the -n option allows you to display a specific number of the most recent log entries. The following command shows the last 50 messages logged:

    $> journalctl -n 50

  • Log messages can be filtered based on their priority using -p option, for example the following command shows only error message from service logind:

    $> journalctl -p error -u logind.service

  • To see kernel message, use:

    $> journalctl -k
$> journalctl -t kernel

  • The -r option displays log entries in reverse chronological order with the latest messages shown first.