systemd debugging

Diagnose boot problems

If your machine gets stuck during startup , First, check that the hang occurs when control is passed to systemd Before or after .

Try not to rhgb and quiet Start the kernel command line . If you see something like this .

• Welcome to Fedora VERSION (codename)!"
• Starting name...
• [ OK ] Started name.

shows systemd Running .

If you can get one shell, Debugging is always easier . If you don't get the login prompt , You can try CTRL+ALT+F__ Switch to another virtual terminal . The problem of displaying server startup may appear as tty1 There is no login on , But other VT Can work .

If the login information is not displayed on any virtual console during startup, it will stop , Before announcing that it must be stuck , Let it retry at most 5 minute . One possibility is that services that are difficult to start will be killed after this timeout , The startup will continue normally . Another possibility is , An important mount point device will not appear , You will see emergency mode .

If do not have shell

If you don't get a normal login , There is no emergency mode shell , You will need to do additional steps to get debugging information from the machine .

• Try CTRL+ALT+DEL restart
• use SysRq Or hard reset to force restart .
• When next started , You'll have to add some kernel command line arguments , It depends on which debugging strategy you choose from the following options .

Debug log to serial console

If you have a hardware serial console , Or you can debug in a virtual machine （ for example , Use virt-manager, You can choose from the menu View -> Text Consoles Switch to the serial port console , Or use virsh console MACHINE Connect from the terminal ）, You can ask for systemd Record a lot of useful debugging information at startup .

systemd.log_level=debug systemd.log_target=console console=ttyS0,38400 console=tty1

Copy

If pid 1 Something goes wrong , The above method is very useful , But if a critical startup service fails later （ Such as network ）, You can configure journald Forward to console .

systemd.journald.forward_to_console=1 console=ttyS0,38400 console=tty1

console= You can specify more than one time ,systemd Will be output to all consoles .

Activate to rescue target or emergency target

Add... To the kernel command line systemd.unit=rescue.target Or just add 1 To start the rescue target directly . If the problem occurs after the basic system is started , Start up " normal " In the process of service , This goal is very useful . If that's the case , You should be able to disable bad services from here . If the rescue target cannot be activated , Smaller emergency targets may be activated .

Add... To the kernel command line systemd.unit=emergency.target or emergency, You can start directly to emergency shell. Please note that , In an emergency , Before editing any file , You must remount the reader / writer of the root file system by yourself .

Common problems that can be solved in a state of emergency are /etc/fstab Problem mount items in . Repair /etc/fstab after , function systemctl daemon-reload, Give Way systemd Refresh its view .

If even the emergency target can't work , You can use it directly init=/bin/sh Boot to shell. If systemd Itself or some of its dependent libraries are corrupted by the file system , This may be necessary . You may need to reinstall the working version of the affected package .

If init=/bin/sh It doesn't work , You must boot from other media .

Open debugging as soon as possible shell

You can enable as early as possible during the startup process shell jurisdiction , In order to take advantage of various systemctl Command diagnostics systemd Related startup problems . Use the following command to enable it

systemctl enable debug-shell.service

Copy

Or specify

Or specify... On the kernel command line systemd.debug-shell=1.

Tips: ： If you find yourself unable to use systemctl And running systemd communicate （ For example, from different startup systems ）, You can specify by --root= To avoid communicating with the manager .

systemctl --root=/ enable debug-shell.service

Copy

Once enabled , The next time you start, you can use CTRL+ALT+F9 Switch to tty9, There is one early in the startup process root shell You can use . You can use this shell To check the status of the service , Read the log , use systemctl list-jobs Find stuck work , wait .

Warning . Warning ： This shell Can only be used for debugging ！ Don't forget to close systemd The function of . After debugging the startup problem , Don't forget to disable systemd-debug-shell.service. Give Way root shell There will be security risks if it is always available .

Can also be kbrequest.target An alias for debug-shell.service, To start the commissioning shell on demand . There are the same security issues , But you can avoid running all the time shell.

Verify prerequisites

There needs to be one （ At least in part ） Filled with /dev. According to your settings （ For example, on embedded systems ）, Check Linux Kernel configuration options CONFIG_DEVTMPFS and CONFIG_DEVTMPFS_MOUNT Whether it is set . in addition , In order to make the operation correct , I suggest that we support cgroups and fanotify, So check Linux Kernel configuration options CONFIG_CGROUPS and CONFIG_FANOTIFY Whether it is set . news "Failed to get D-Bus connection: In all kinds of systemctl In operation , appear "Failed to get D-Bus connection: No connection to service manager. " A hint of , These options are missing .

If there is shell

When systemd Run to provide you with shell Degree of , Please use it to extract useful information for debugging . Start with these parameters on the kernel command line .

systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M printk.devkmsg=on In order to improve the systemd The degree of rudeness , Give Way systemd Write logs to the kernel log buffer , Increase the size of the kernel log buffer , And prevent the kernel from dropping information . arrive shell after , Take a look at the log .

When reporting a bug when , Pipe it to a file , And attach it to bug In the report .

Check work that may be stuck , Please use .

Is listed as " Running " My homework is in " wait for " Must be completed before the job is allowed to start execution .

Diagnose shutdown problems

Just like the boot problem , When you encounter a hang during shutdown , Make sure you at least wait 5 minute , To distinguish between a persistent hang and a bad service that just times out . Then what is worth testing , Whether the system is correct CTRL+ALT+DEL Any reaction .

If your system shuts down （ Whether it is restart or power failure ） Stuck , First, test whether the kernel itself can use these commands to force a reboot or power down .

If either of these commands doesn't work , That's probably the kernel problem , instead of systemd The problem of .

The shutdown is finally completed

If the normal restart or shutdown works , But it took suspiciously long , that

• Start with debug options .

systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M printk.devkmsg=on enforcing=0

Copy

• Save the following script as /usr/lib/systemd/system-shutdown/debug.sh And make it executable .

mount -o remount,rw /
dmesg > /shutdown-log.txt
mount -o remount,ro /

Copy

• Restart

Look for records in the results file shutdown-log.txt Timeout in , and / Or attach it to bugreport in .

Shutdown never completed

If a normal restart or shutdown is never completed even after a few minutes , Then the above method of creating shutdown log will not help , You must use other methods to get logs . There are two options that are useful for debugging startup issues and also for shutdown issues .

• Use the serial console
• Use debug shell-- Not only can it be used from an early start , And it is active until the late shutdown .

Status and logs of services

When the service fails to start ,systemctl Will give you a generic error message ：

Job failed. See system journal and 'systemctl status' for details.

Copy

The service may have printed its own error message , But you don't see , Because the systemd The running service is independent of your login session , Their output is not connected to your terminal . But this does not mean that the output is lost . By default , Service stdout and stderr All point to systemd journal , Service pass syslog(3) The generated log will also enter the log ,systemd It also saves the exit code of the failed service . Let's see .

foo.service - mmm service
          Loaded: loaded (/etc/systemd/system/foo.service; static)
          Active: failed (Result: exit-code) since Fri, 11 May 2012 20:26:23 +0200; 4s ago
         Process: 1329 ExecStart=/usr/local/bin/foo (code=exited, status=1/FAILURE)
          CGroup: name=systemd:/system/foo.service
May 11 20:26:23 scratch foo[1329]: Failed to parse config

Copy

In this case , The service to PID by 1329 The process runs , And with an error code 1 sign out . If you are in root or adm Run as the user of the group systemctl status, You will get a few lines from the logs written by the service . In this case , The service only generates an error message .

To list logs , Please use journalctl command .

If you have one syslog service （ such as rsyslog） Running , The log will also forward information to it , So you will be in /var/log/messages Find them in （ Depending on rsyslog Configuration of ）.