Debugging systemd Using systemctl

To debug failed services using systemctl:

1. List failed services in the system:

   $> sudo systemctl --failed
   

   You will get a list of failed services. To see a list of all the services in the system, use the command:

   $> sudo systemctl list-unit-files --type=service
   

2. Check status of the failed service for more detailed information:

   $> sudo systemctl status <service_name>.service
   

3. See the service logs for more insight:

   $> sudo journalctl -b -u <service_name>.service
   

4. You can further increase the log level to get debug level information:

   $> sudo systemctl log-level debug
   

   Reload the systemd daemon and restart a service:

   $> sudo systemctl daemon-reload
   $> sudo systemctl restart  <service_name>.service
   

   Now you can see debug level information in the service log.

5. You can also attach strace with the service daemon to see system call and signal status:

   • Get the PID of the main process from service status. It is listed as Main PID:.

   • Attach strace with the PID:

     $> sudo strace -f -s 100 -p <Main_PID>
     

6. Retune the service configuration in runtime:

   $> systemctl edit --runtime <service_name>.service
   

   • Uncomment the [Service] section and the configuration you want to enable or disable. You can add any new configuration. This basically overrides your base configuration.

   • Save the configuration as /run/systemd/system/<service_name>.d/override.conf.

   • Reload the systemd daemon and restart the service as mentioned in step 4.

   • You can check if your service is using the new configuration with the command:

     $> sudo systemctl show <service_name>.service
     

   • For checking the base configuration:

     $> sudo systemctl cat <service_name>.service
     

7. If the new configuration works for you, you can check the exposure level of the service using the command:

   $> systemd-analyze security
   $> systemd-analyze security <service_name>.service #For detailed information
   

8. Update the configuration in the Ghaf repository and build it. Hardened service configs are available in the directory ghaf/modules/common/systemd/hardened-configs.