Tips & tricks for systemd

Here are few tips I discovered while writing few systemd units for my new Debian Jessie system. You can find my unit files in my GitHub repository.

Perl and stdout in systemd unit

I convered my simple perl script to systemd. Before, it was running in a screen session and outputting some status information to stdout. I used StandardOutput=journal parameter to redirect stdout to systemd’s journal. However the journal did not contain any output of that script. I have found out that by redirecting perl’s stdout to anything other than a terminal, perl turns on buffering, so the output would appear only after 8 kB of text. To disable the buffering, just put $|=1; in the beginning of your script. You can display the unit’s output by using command journalctl -u myunit.

Handing of /var/run and /run subdirs

Note: /var/run is being replaced by (and symlinked from) /run, which is a tmpfs filesystem on modern distros. That implies it is empty after each boot.

When a daemon is run under non-root user, its init.d script traditionally created a subdir inside the /var/run, which was then made owned by the non-root user under which the daemon will be run. Here is how the same can be accomplished in a systemd unit file.

[Unit]
Description=DCC (Distributed Checksum Clearinghouses) interface daemon
Before=spamassassin.service

[Service]
Type=forking
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /run/dcc
ExecStartPre=/bin/chown -R vscan:vscan /var/run/dcc
ExecStart=/var/dcc/libexec/dccifd
User=vscan

We use ExecStartPre to create the directory before the actual daemon is run. When PermissionsStartOnly is true, only the actual daemon is run under the defined user, but the pre-start commands are run under root. I use mkdir -p, which doesn’t return error even when the directory already exists (for example when the daemon is restarted several times).

UPDATE 13.07.2016: Better solution is to use system’d options RuntimeDirectory and RuntimeDirectoryMode (see man systemd.exec). However this can’t be used when the daemon itself changes the UID to unpriviledged and so there is no User option in the unit’s config file. The above example could be re-written as:

[Unit]
Description=DCC (Distributed Checksum Clearinghouses) interface daemon
Before=spamassassin.service

[Service]
Type=forking
RuntimeDirectory=dcc
ExecStart=/var/dcc/libexec/dccifd
User=vscan

Making your own units to start automatically at system boot

Systemd uses so-called targets instead of run levels. The main difference is that you can have only one run level active at a time. Targets are states which can be activated (or better said achieved), but there is nothing like an active target and more than one target can be achieved at the same time. The most interesting target is the multi-user.target.

To start your service at boot, add these lines to your unit’s config file.

[Install]
WantedBy=multi-user.target

However, by default the unit is in disabled state, so it won’t be started. To enable it, run systemctl enable myunit. (This will make a symlink from /etc/systemd/system/multi-user.target.wants/myunit.service to /etc/systemd/system/myunit.service.)

How to modify existing unit file (and set open files limit for MySQL)

By default, systemd units shipped with distro packages are put into /lib/systemd/system directory. Don’t modify any of these files as your changes would get lost on update.

If you want to replace whole unit with your own version (and loose any further changes by updated package), you can put your copy of the unit file in /etc/systemd/system. When this exists, systemd will use it instead of the one in /lib.

If you want just to change few settings, create a new directory /etc/systemd/system/mysql.service.d (in our case for MySQL daemon). Each file there will be read after the main unit file in /lib. I have created a file override.conf with this content to stop MySQL complaining about not enough open files limit.

[Service]
LimitNOFILE=32000

This is the warning message MySQL produces when the open files limit is too low for your configuration:

[Warning] Changed limits: max_open_files: 1024 (requested 5000)

[Warning] Changed limits: table_open_cache: 431 (requested 2000)

Certain options can be overwritten by specifying them 2nd time, some options are additive, so specifying them multiple times just adds new values to the list. If you want to replace such option, specify it without arguments first (Option=).