Systemd Mechanics

From trapsink.com
Jump to: navigation, search


Systemd components.png

System bootup process with systemd


Overview

From the systemd project homepage:

systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux cgroups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic. It can work as a drop-in replacement for sysvinit.


Configuration

File / Directory Function
/etc/hostname The host name for the system (yes, it's part of systemd now)
/etc/os-release Standardization of the various distribution ID files like /etc/redhat-release and similar
/etc/vconsole.conf Configuration of the default keyboard mapping and console font
/etc/locale.conf Configuration of the system-wide locale
/etc/machine-id Machine ID file, superseding D-Bus' machine ID file. Guaranteed to exist and be valid on a systemd system
/etc/machine-info Metadata about a host, like a pretty host name and an icon name; maintained by systemd-hostnamed
/etc/systemd/system.conf Read when systemd is run as a system instance
/etc/systemd/user.conf Read when systemd is run as a non-system service
/etc/systemd/journald.conf Configures the systemd-journald system service
/etc/systemd/logind.conf Configures various parameters of the systemd login manager systemd-logind
/etc/modules-load.d/*.conf Drop-in directory for kernel modules to statically load at boot
/etc/sysctl.d/*.conf Drop-in directory for kernel sysctl parameters, extending /etc/sysctl.conf
/etc/tmpfiles.d/*.conf Drop-in directory for configuration of runtime files that need to be removed/created/cleaned up at boot and during uptime
/etc/binfmt.d/*.conf Drop-in directory for registration of additional binary formats for systems like Java, Mono and WINE
/etc/systemd/ Main configuration directory for customizations
/var/log/journal/ Location of the systemd-journald persistent log data
/usr/lib/systemd/ Main directory of distro-provided systemd components
/var/lib/systemd/ Main directory of runtime systemd components


Common Tools

Primary commands used in the day-to-day systemd world:

Command Purpose
systemctl Used to introspect and control the state of the systemd system and service manager
journalctl Query the contents of the systemd journal as written by systemd-journald.service
hostnamectl Used to query and change the system hostname and related settings
timedatectl Query and change the system clock and its settings
localectl Control the system locale and keyboard layout settings
loginctl Control the systemd login manager systemd-logind
systemd-cat Connect a pipeline or program's output with the journal (like logger)
systemd-cgls Recursively shows the contents of the selected Linux control group hierarchy in a tree
systemd-cgtop Show top control groups by their resource usage; useful when using cgroups to limit resources
systemd-delta Identify and compare configuration files in /etc that override default counterparts in /usr
kernel-install Add and remove kernel and initramfs images to and from /boot


Boot Parameters

On boot systemd activates (by default) the target unit default.target whose job is to activate services and other units by pulling them in via dependencies. To override the unit to activate, systemd parses its own kernel command line arguments via the systemd.unit= command line option. This may be used to temporarily boot into a different boot unit. The classic runlevels are replaced as following:

Parameter Purpose
systemd.unit=rescue.target A special target unit for setting up the base system and a rescue shell (similar to runlevel 1)
systemd.unit=emergency.target Very similar to passing init=/bin/sh but with the option to boot the full system from there
systemd.unit=multi-user.target Setting up a non-graphical multi-user system
systemd.unit=graphical.target Setting up a graphical login screen


Parameter Type Default Purpose
systemd.unit= string default.target Overrides the unit to activate on boot. Example: rescue.target or emergency.target
systemd.dump_core= boolean true If true systemd dumps core when it crashes. Otherwise no core dump is created
systemd.crash_shell= boolean false If true systemd spawns a shell when it crashes. Otherwise no core dump is created
systemd.crash_chvt= int -1 If positive systemd activates the specified virtual terminal when it crashes
systemd.confirm_spawn= boolean false If true asks for confirmation when spawning processes
systemd.show_status= boolean true If true shows terse service status updates on the console during bootup
systemd.sysv_console= boolean true If true output of SysV initscripts will be directed to the console
systemd.log_target= string journal-or-kmsg Set log target. One of: console, journal, syslog, kmsg, journal-or-kmsg, syslog-or-kmsg, null
systemd.log_level= int, constant info Set log level. Numerical log level or one of: emerg, alert, crit, err, warning, notice, info, debug
systemd.log_color= boolean true Highlight important log messages
systemd.log_location= boolean true Include code location in log messages. This is mostly relevant for debugging purposes

For details about these special systemd boot units, view the systemd.special man page.


Runlevels/Targets

Systemd has a concept of targets which serve a similar purpose as runlevels but act a little different. Each target is named instead of numbered and is intended to serve a specific purpose. Some targets are implemented by inheriting all of the services of another target and adding additional services to it. There are systemd targets that mimic the common sysvinit runlevels so you can still switch targets using the familiar telinit RUNLEVEL command. The runlevels that are assigned a specific purpose on vanilla systemd-enabled RHEL/Fedora installs; 0, 1, 3, 5, and 6; have a 1:1 mapping with a specific systemd target.

SysVinit Runlevel systemd Target Notes
0 runlevel0.target, poweroff.target Halt the system.
1, s, single runlevel1.target, rescue.target Single user mode.
2, 4 runlevel2.target, runlevel4.target, multi-user.target User-defined/Site-specific runlevels. By default, identical to 3.
3 runlevel3.target, multi-user.target Multi-user, non-graphical. Users can usually login via multiple consoles or via the network.
5 runlevel5.target, graphical.target Multi-user, graphical. Usually has all the services of runlevel 3 plus a graphical login.
6 runlevel6.target, reboot.target Reboot
emergency emergency.target Emergency shell


Manipulating Runlevels

SysVinit Command systemd Command Notes
grep initdefault /etc/inittab systemctl list-units --type=target Show current runlevel
telinit 3 systemctl isolate multi-user.target (or)
 systemctl isolate runlevel3.target (or)
 telinit 3
Change to multi-user runlevel
vi /etc/inittab
 (change initdefault)
systemctl enable multi-user.target --force (or)
ln -sf /lib/systemd/system/multi-user.target \
  /etc/systemd/system/default.target
Set to use multi-user runlevel on next reboot


Manipulating Services

Note that multiple service can be specified after the command, so `systemctl start foo.service bar.service baz@foo.service` is a valid command.

SysVinit Command systemd Command Notes
service foobar start systemctl start foobar.service Used to start a service (not reboot persistent)
service foobar stop systemctl stop foobar.service Used to stop a service (not reboot persistent)
service foobar restart systemctl restart foobar.service Used to stop and then start a service
service foobar reload systemctl reload foobar.service When supported, reloads the config file without interrupting pending operations.
service foobar condrestart systemctl condrestart foobar.service Restarts if the service is already running.
service foobar status systemctl status foobar.service Tells whether a service is currently running.
ls /etc/rc.d/init.d/ systemctl list-unit-files --type=service (or)
 ls /lib/systemd/system/*.service \
   /etc/systemd/system/*.service
Used to list the services that can be started or stopped
Used to list all the services and other units
chkconfig foobar on systemctl enable foobar.service Turn the service on, for start at next boot, or other trigger.
chkconfig foobar off systemctl disable foobar.service Turn the service off for the next reboot, or any other trigger.
chkconfig foobar systemctl is-enabled foobar.service Used to check whether a service is configured to start or not in the current environment.
chkconfig --list systemctl list-unit-files --type=service (or)
 ls /etc/systemd/system/*.wants/
Print a table of services that lists which runlevels each is configured on or off
chkconfig foobar --list ls /etc/systemd/system/*.wants/foobar.service Used to list what levels this service is configured on or off
chkconfig foobar --add systemctl daemon-reload Used when you create a new service file or modify any configuration

All /sbin/service and /sbin/chkconfig lines listed above continue to work on Fedora (and likely RHEL7), and will be translated to native equivalents as necessary. The only exception is chkconfig --list.

In SysVinit, services can define arbitrary commands. Examples would be service iptables panic, or service httpd graceful. Native systemd services do not have this ability; any service that defines an additional command in this way would need to define some other - service-specific - way to accomplish this task when writing a native systemd service definition. Check the package-specific release notes for any services that may have done this.


Customizing Services

The best way to customize unit files is to add /etc/systemd/system/foobar.service.d/*.conf where foobar.service.d is the name of the service you want to customize. If a directory doesn't already exist, create one and add a conf file with the settings you want to override.

Example: raise the number of Open Files for MariaDB (previously configured in /etc/security/limits.conf):

/etc/systemd/system/mariadb.service.d/limits.conf
[Service]
LimitNOFILE=10000
LimitMEMLOCK=100000

Example: automatically restart Apache if it dies/killed/etc.:

/etc/systemd/system/httpd.service.d/restart.conf
[Service]
Restart=always
RestartSec=30

Alternatively, you can copy the distribution provided unit file from /lib/systemd/system to /etc/systemd/system since the latter has higher precedence. If a line starts with .include followed by a file name, the specified file will be parsed at this point.

/etc/systemd/system/httpd.service
.include /lib/systemd/system/httpd.service
[Service]
Nice=-5

Don't forget to reload systemd daemon using systemctl daemon-reload and systemctl restart foobar after editing a unit file where foobar is the name of the unit. Also note that you can systemd-delta to list the unit files which have been customized and also the precise differences

Be careful when using .include together with directives that can be defined multiple times like EnvironmentFile, since we can only add new directives, but we can't remove already defined ones. You have to copy the whole file from /lib/systemd/system to /etc/systemd/system in this case


Instanced Units (.service, .socket, etc...)

systemd allows you to set up instanced services, or services that you can run multiple times in parallel with a small change to the unit file, such as getty, dhcpcd, sshd, and more. The way you define these services is slightly different from normal service files such as sshd.service or foobar.socket. Instead of name.type, you would call the file name@.type, and this would become an instanced unit file. The internals of the unit file are also somewhat different:

/usr/lib/systemd/system/dhcpcd@.service
[Unit]
Description=dhcpcd on %I
Wants=network.target
Before=network.target
BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device

[Service]
Type=forking
PIDFile=/run/dhcpcd-%I.pid
ExecStart=/usr/bin/dhcpcd -q -w %I
ExecStop=/usr/bin/dhcpcd -x %I

[Install]
WantedBy=multi-user.target
# # TO run the service you would use:
# systemctl start dhcpcd@eth1.service
# # enabling would also be the same, unit@instance.type
# systemctl enable dhcpcd@eth1.service

These commands would start or enable the dhcpcd service with the instance 'eth1'. All of the %i and %I variables would be replaced with the instance, in this case eth1. This allows you to have a generic service file for dhcpcd or other units, instead of writing a new service file for each instance of the daemon.

The %i and %I are two of the different ways to specify the instance name in a service file. You can see more here: systemd.unit


Path Units (.path)

systemd includes many types of unit files. The .path unit file allows systemd to watch a path with inotify and execute a service according to the .path file. You can find a full manual here: systemd.path


Mount Units (.mount)

Most mounts are specified in /etc/fstab and the systemd-generator creates the mount units dynamically upon boot, however there are cases where it does not work as expected or needed. One of those types is a bind mount - a bind mount needs to have a unique mount file created for it outside of fstab. See the upstream systemd.mount documentation for full details on all possible options and features that can be used.

Example Bind Mount - /var/tmp to /tmp

One of the common bind mounts to create is for /var/tmp to /tmp as it is a line item in the CIS Hardening Guidelines as a requirement. The name of the mount unit file must be the name of the destination with the leading slash removed, and all internal slashes converted to hyphens; it is placed in the expected runtime location if created by hand (/etc/systemd/system) and not delivered via a package (/usr/lib/systemd/system).

/etc/systemd/system/var-tmp.mount
# https://www.freedesktop.org/software/systemd/man/systemd.mount.html
# -> Automatic Dependencies
[Unit]
DefaultDependencies=yes
 
[Mount]
What=/tmp
Where=/var/tmp
Type=none
Options=bind
 
[Install]
WantedBy=local-fs.target

The unit must be enabled as per any other unit type:

# systemctl daemon-reload
# systemctl enable var-tmp.mount

With this particular bind mount, if anything is currently using /var/tmp those services must be stopped first to activate (systemctl start ...) the unit. It is possible a full reboot is needed depending on which services are using the directory.


Service Resource Limits

The integration of cgroups into systemd allows us to limit resource usage for a service; the limiting of CPU, Memory and Block I/O are the primary concern to most system administrators. These limits are a per-cgroup limit, not per-process, so for Apache it would only allow the whole cgroup to use 1G of memory, no matter how many tiny processes it spawns.

By default all processes get an even value of CPU time (1024); by raising this number per-service more CPU shares are given over those at the default 1024 level:

/etc/systemd/system/httpd.service
.include /lib/systemd/system/httpd.service
[Service]
CPUShares=1500

Memory is limited in the exact same way; this setting understands K, M, G, T suffixes (base 1024)

/etc/systemd/system/httpd.service
.include /lib/systemd/system/httpd.service
[Service]
MemoryLimit=1G

Block I/O is weighted; default weight is 1000, valid range is from 10 to 1000 and can be used by service, by block device or by named directory:

/etc/systemd/system/httpd.service
.include /lib/systemd/system/httpd.service
[Service]
BlockIOWeight=500
BlockIOWeight=/dev/disk/by-id/dm-name-vg_local-lv_home 250
BlockIOWeight=/var/www/html 750


Service Types

Type Definition
simple (default) systemd considers the service to be started up immediately. The process must not fork. Do not use this type if other services need to be ordered on this service, unless it is socket activated.
forking systemd considers the service started up once the process forks and the parent has exited. Specify PIDFile= as well so systemd can keep track of the main process
oneshot Useful for scripts that do a single job and then exit. Set RemainAfterExit=yes as well so that systemd still considers the service as active after the process has exited
notify Identical to Type=simple, but with the stipulation that the daemon will send a signal to systemd when it is ready
dbus The service is considered ready when the specified BusName appears on DBus's system bus


Operational Commands

Command Purpose
systemctl --failed List failed units only
systemctl list-jobs Look for jobs "running" (boot waits for completion) and "waiting" (executed only after those which are "running" are completed)
systemctl list-units -t service --all List all available services and their current status
systemctl list-units -t service Show all active services
systemctl status foobar.service Examine the current runtime status of a service
systemctl list-units -t target --all Show all available targets.
systemctl list-units -t target Show all active targets
systemctl show -p "Wants" multi-user.target See which services a target pulls in
systemctl kill httpd.service Send the kill (SIGTERM) signal to all processes of a service
systemctl kill -s SIGKILL httpd.service Send the specified signal to all processes of a service
systemctl kill -s HUP --kill-who=main httpd.service Send the HUP signal to only the parent process of the service
systemd-analyze blame List systemd unit initialization times at boot
systemd --test --system --unit=multi-user.target Examine what gets started when when booted into a specific target


Power Management

Command Purpose
systemctl reboot Shut down and reboot the system
systemctl poweroff Shut down and power-off the system
systemctl suspend Suspend the system
systemctl hibernate Put the system into hibernation
systemctl hybrid-sleep Put the system into hybrid-sleep state (or suspend-to-both)


Journal Commands

Compatibility with classic syslog implementations is provided via a socket /run/systemd/journal/syslog, to which all messages are forwarded. To make the syslog daemon work with the journal, it has to bind to this socket instead of /dev/log.

Command Purpose
journalctl -b -0 Show all messages from the current boot (-1 = previous, -2 = two boots ago, etc.)
journalctl -b -p err Show only priority ERROR level boot log messages
journalctl --since=yesterday Show only logs since yesterday if rebooting is seldom (servers)
journalctl -f Follow the journal like tail -f /var/log/messages
journalctl /usr/sbin/httpd Show all messages by a specific binary
journalctl /usr/sbin/vpnc /usr/sbin/dhclient Show messages interleaved between two binaries
journalctl _PID=1234 Show all messages by a specific process ID
journalctl -u httpd Show all messages by a specific unit
journalctl -u httpd --since=00:00 --until=9:30 Show all messages for a unit and timeframe
journalctl _TRANSPORT=kernel Show kernel ring buffer
journalctl /dev/sdc Show messages related to a specific device


Notes

  • RHEL7 will not have kdbus, which is going to make system* --user require $DISPLAY variable because of dbus. kdbus is something that has been added very recently to the kernel and systemd.


References