Systemd - SolarisJapan/lunaris-wiki GitHub Wiki

Systemd Services

Systemd is a program that controls what runs at startup in a Linux system. It can also monitor the processes started in a systemd service and allow for easy logging, controls such as restart / stop, and can restart the process if it crashes.

Creating a service

Service Unit Files

The scripts for systemd services are written into unit files. Here is an example unit file for starting Sidekiq via service. Note that when using RVM the ExecStart command may need to be run using RVM's wrappers (exact directory may depend on RVM version and use of user gemsets).

# May need to use this:
ExecStart=/usr/local/rvm/wrappers/ruby-2.6.1@global/bundle exec sidekiq -e production
# Rather than the suggested:
# ExecStart=/usr/local/bin/bundle exec sidekiq -e production

Service Permissions

There are two main types of services in the scope of permissions: system and user. If the server is set up to have multiple users involved in deployment or maintenance of the app with differing levels of permissions then it may be beneficial to split services into the appropriate category. Often guides will suggest creating a deploy user without super user access and this would be a good use case for a user service.

If using user services be sure to run the following command and relog into the given user if applicable.

loginctl enable-linger $USER

Store unit files here:

# System services
/etc/systemd/system

# User services
/etc/systemd/user

For user unit files it seems that the attribute WantedBy=multi-user.target (as used in the Sidekiq example above) may not be applicable. Instead user unit files may need to use WantedBy=default.target or completely remove this attribute.

[Install]
#WantedBy=multi-user.target
WantedBy=default.target

Purpose

The most obvious reasons is so that the app / script will restart on crash or server reboot. As seen in the Sidekiq example, restarting after a crash is set within the unit file. In order to have the service start on server reboot use the following command:

systemctl [--user] enable example.service

Systemd also provides realtime monitoring of the service's output (explained below). It can also help to run apps which may have multiple self-contained components. In order to ensure these operate together you may want to look into systemd targets. This can allow you to group unit files so that they can all be started or stopped simultaneously. They can also be set to exit if the other members of the target are not active for example.

Commands

Management

Systemd has commands to help manage and monitor services.

# Manage the service
systemctl [--user] start|stop|restart example.service

# Check that the service is running correctly
systemctl [--user] status example.service

# Reload all unit files after changes
systemctl [--user] daemon-reload

Monitoring

Systemd works with journalctl to allow you to see logs output by the script being run.

# All services
journalctl [--follow]

# Specific system service
journalctl [--follow] -u example.service

# Specific user service
journalctl [--follow] --user-unit example_user.service

Targets

If your service unit files have been grouped into a target they can be managed in the same way as service unit files. Exactly what it means to start or stop your target depends on whether the services are "required", "wanted", etc as set in the target unit file.

systemctl [--user] start|stop|restart example.target

Troubleshooting

Failed to connect to bus

If using user services and experiencing this issue

user@server:~# systemctl --user status
Failed to connect to bus: No such file or directory

Try running this command again and then relogging into the user.

loginctl enable-linger $USER

If the problem persists check the value of this environment variable:

# Should look something like
# /run/user/1000
printenv XDG_RUNTIME_DIR

If this environment variable is not set you can try adding it to your environment:

export XDG_RUNTIME_DIR=/run/user/$(id -u)

Permissions

If there seem to be issues with permissions then it could be a good idea to use chown to grant ownership of the specific user unit file to the appropriate user.

If journalctl is giving permissions errors for user unit files it provides some solutions.