LinuxSeptember 8, 2024

A field guide to writing systemd service units

systemd attracts strong opinions, and I held a few of the loud ones for years. Then I spent a weekend replacing a pile of brittle init scripts with unit files and quietly changed my mind. A good unit file does in fifteen lines what those scripts did in two hundred, and it does it more reliably. Here is the subset I actually use.

The minimal unit that does the right thing

Most services need fewer directives than you would guess. This is a complete, production-ready unit for a typical long-running daemon:

# /etc/systemd/system/widgetd.service
[Unit]
Description=Widget processing daemon
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/widgetd --config /etc/widgetd.toml
Restart=on-failure
RestartSec=5
User=widget
Group=widget

[Install]
WantedBy=multi-user.target

That is enough for systemd to start the service on boot, run it as an unprivileged user, and restart it if it crashes. Everything beyond this is refinement, not foundation.

Get the Type right

The single most common mistake is the wrong Type. It tells systemd how to know your service has actually started:

Type=simple — the process you exec is the service and stays in the foreground. This is the default and what you want most of the time.
Type=exec — like simple, but systemd waits until the binary has actually been executed before considering startup complete. A safer simple.
Type=notify — the service tells systemd it is ready via sd_notify(). Use this when "started" means "finished warming up," not just "process exists."
Type=forking — the process daemonizes and the parent exits. Avoid it for anything you write yourself; foreground processes are simpler to supervise.

If your service forks into the background and you leave Type=simple, systemd thinks it died the instant the parent exits and may restart it in a loop. Match the type to the behavior.

Restart policy that does not mask bugs

Auto-restart is a feature and a footgun. Restart=always will happily restart a service that crashes on startup, hiding a broken deploy behind an endless loop. I prefer on-failure with rate limiting, so a service that cannot start fails visibly instead of thrashing forever:

[Service]
Restart=on-failure
RestartSec=5
# Give up if it fails 5 times within 60 seconds
StartLimitIntervalSec=60
StartLimitBurst=5

When the burst limit trips, the unit enters a failed state and stops trying. That is exactly what you want: a clear signal in systemctl status rather than a service that quietly burns CPU restarting a broken binary all night.

Sandboxing for almost free

This is the part that won me over. systemd can sandbox a service with a few directives, no code changes, no container. Each line below removes a class of attack:

[Service]
# Service gets a private /tmp, isolated from the host
PrivateTmp=true
# Most of the filesystem is read-only to the service
ProtectSystem=strict
# Home directories are invisible
ProtectHome=true
# Explicitly allow the one path it must write
ReadWritePaths=/var/lib/widgetd
# Cannot gain new privileges via setuid binaries
NoNewPrivileges=true

A service that only needs to read its config and write one data directory has no business being able to touch /etc or a user's home. These directives enforce that for free. Run systemd-analyze security widgetd.service afterward — it scores your unit and lists exactly what is still exposed. It is an oddly motivating little report card.

Reading the logs

Because the service runs in the foreground under systemd, its stdout and stderr go straight to the journal. No log file configuration required:

# Live tail for one unit
journalctl -u widgetd -f

# Just this boot, with timestamps
journalctl -u widgetd -b

# Since a relative time
journalctl -u widgetd --since "30 min ago"

The workflow

After editing any unit, reload the systemd manager configuration before acting on it. Forgetting this step is responsible for an embarrassing share of "my change did nothing" moments, including several of mine:

systemctl daemon-reload
systemctl enable --now widgetd
systemctl status widgetd

What I skip

I avoid Type=forking in anything I control, I do not reach for EnvironmentFile when a couple of Environment= lines will do, and I never use Restart=always on a service that has a meaningful startup phase. Keep the unit small, let the sandbox do the heavy lifting, and let the journal be your log file. That is most of what I have learned, and it has held up well.

Filed under Linux. Corrections welcome via the about page.