Services and systemd

I spent some time last week poking around systemd and trying to figure out how certain things work. I can't claim to be an expert yet, but I did uncover some things that I found to be very useful.

If you try to start a service on a machine with systemd (Fedora-15, for instance), it actually looks different then with a traditional SysV style init.

SysV:

[root@localhost ~]# service mongod start
Starting mongod:                                           [  OK  ]
[root@localhost ~]# service mongod stop
Stopping mongod:                                           [  OK  ]
[root@localhost ~]# /etc/init.d/mongod start
Starting mongod:                                           [  OK  ]
[root@localhost ~]# /etc/init.d/mongod stop
Stopping mongod:                                           [  OK  ]
[root@localhost ~]# 

Systemd:

[root@localhost ~]# service mongod start
Starting mongod (via systemctl):                           [  OK  ]
[root@localhost ~]# service mongod stop
Stopping mongod (via systemctl):                           [  OK  ]
[root@localhost ~]#  /etc/init.d/mongod start
Starting mongod (via systemctl):                           [  OK  ]
[root@localhost ~]#  /etc/init.d/mongod stop
Stopping mongod (via systemctl):                           [  OK  ]
[root@localhost ~]# 

"(via systemctl)" is a small but important change to how services are launched. With SysV-style scripts, the scripts are executed more-or-less directly from the bash shell they are launched from (the "service" binary does a little more in terms of cleaning up the environment, but it still ends up exec'ing the script in the end).

With systemd this all changes. One of the first things nearly all initscripts do is to source /etc/init.d/functions. On a systemd-enabled system, the very first thing that /etc/init.d/functions does is to execute systemctl and then exit (ignoring the rest of the initscript). What systemctl does is to put a message on dbus asking for the service you specified to be started. systemd itself is listening on dbus; when it sees a message like this, it picks up the message and proceeds to act on it. It first looks to see if there is a native systemd unit file for this service; if there is, it starts the service according to the native unit file and returns status to systemctl, which returns status to service. If there is no native systemd unit file, it then looks in /etc/init.d for a legacy script. If one is found, then it fork and exec's that script, and returns the status to systemctl, which returns to service.

This leads to one of the most visible issues with systemd, in that there is no output if the initscript fails. That is, if you are using a legacy style initscript and you are used to certain output being shown when something fails, it may not be shown anymore since that output was consumed by systemd itself and not returned to systemctl.

One way to deal with this is to skip the redirect from service to systemctl. There are different ways to do this depending whether you are using the "service" binary or if you are executing the script directly. If you are using the service binary, it understands a new flag to skip redirection to systemd:

service --skip-redirect foo start

If you are directly executing the initscript, you need to pass an environment variable:

SYSTEMCTL_SKIP_REDIRECT=1 /etc/init.d/foo start