Controlling Galaxy with systemd or Supervisor

name: inverse
layout: true
class: center, middle, inverse

</span></div>

</span></div>

---

# Controlling Galaxy with systemd or Supervisor

<div class="contributors-line">
		Authors: 
<a href="/training-material/hall-of-fame/natefoo/" class="contributor-badge contributor-natefoo"><img src="https://avatars.githubusercontent.com/natefoo?s=27" alt="Avatar">Nate Coraor</a>

<a href="/training-material/hall-of-fame/hexylena/" class="contributor-badge contributor-hexylena"><img src="/training-material/assets/images/orcid.png" alt="orcid logo"/><img src="https://avatars.githubusercontent.com/hexylena?s=27" alt="Avatar">Helena Rasche</a>

</div>

<div class="footnote" style="bottom: 8em;"><i class="far fa-calendar" aria-hidden="true"></i><span class="visually-hidden">last_modification</span> Updated: Apr 6, 2021</div>

<div class="footnote" style="bottom: 6em;">

<i class="far fa-play-circle" aria-hidden="true"></i><span class="visually-hidden">video-slides</span> <a href="/training-material/videos/watch.html?v=/admin/tutorials/systemd-supervisor/slides">Video slides</a> |

<i class="fas fa-file-alt" aria-hidden="true"></i><span class="visually-hidden">text-document</span><a href="slides-plain.html"> Plain-text slides</a>
</div>

<div class="footnote" style="bottom: 2em;">
    <strong>Tip: </strong>press <kbd>P</kbd> to view the presenter notes
    | <i class="fa fa-arrows" aria-hidden="true"></i><span class="visually-hidden">arrow-keys</span> Use arrow keys to move between slides

</div>

???
Presenter notes contain extra information which might be useful if you intend to use these slides for teaching.

Press `P` again to switch presenter notes off

Press `C` to create a new window where the same presentation will be displayed.
This window is linked to the main window. Changing slides on one will cause the
slide to change on the other.

Useful when presenting.

---

class: special

# systemd

The current Linux init system

Used to bootstrap the user space and to manage system processes after booting

???

- systemd is a popular linux init system.
- It manages processes and ensures they are running after boot.
- It can restart processes when they crash.

---
class: left

# systemd

Replaces:
- `/etc/init.d/*`, `/etc/rc?.d`
- `service` command
- Upstart
- `update-rc.d`

INI config format

???

- systemd replaces many previous alternative init systems
- it is quite easy to work with

---
# systemd - Layout

- `/lib/systemd/system` - System service definitions
- `/etc/systemd/system` - User-defined service definitions

???

- systemd has two separate pools of service definitions.
- in lib are the system and package provided definitions.
- in etc are the user provided definitions.
- Galaxy's will be in /etc.

---
# systemd - Galaxy service unit

```ini
[Unit]
Description=Galaxy
After=network.target
After=time-sync.target

[Service]
UMask=022
Type=simple
User=galaxy
Group=galaxy
WorkingDirectory=/srv/galaxy/server
TimeoutStartSec=10
ExecStart=/srv/galaxy/venv/bin/uwsgi --yaml /srv/galaxy/config/galaxy.yml --stats 127.0.0.1:4010
Environment=HOME=/srv/galaxy VIRTUAL_ENV=/srv/galaxy/venv PATH=/srv/galaxy/venv/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin DOCUTILSCONFIG= PYTHONPATH=/srv/galaxy/server/lib/galaxy/jobs/rules
MemoryLimit=32G
Restart=always

MemoryAccounting=yes
CPUAccounting=yes
BlockIOAccounting=yes

[Install]
WantedBy=multi-user.target
```

???

- Here is an example systemd service definition, or a unit.
- At the top section, it describes the service name, and any dependencies.
- For galaxy, we only want it to start when the network is available.
- Next is the service section.
- here we define that it is a simple service, run by the user and group galaxy.
- it starts in a particular working directory.
- and if it does not start within 10 seconds, something is wrong.
- We provide a command that should be run.
- And some environment variables like the path or home directory.
- A memory limit is provided, which systemd enforces through cgroups.
- next, the process should always restart if it crashes.
- And various accounting is enabled.

---
# systemd - systemctl

`systemctl` is the command line interface to systemd

- `systemctl status <name>[.service]`
- `systemctl <start|stop|restart|...> <name>[.service]`
- `systemctl <enable|disable> <name>[.service]`

???

- the systemctl command lets you interact with the service.
- You can start, stop, or restart the service.
- You can enable or disable the service if you do or do not want it to start at boot.

---
# systemd - Galaxy service

```console
$ systemctl status galaxy
● galaxy.service - Galaxy
   Loaded: loaded (/etc/systemd/system/galaxy.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-02-26 19:46:03 UTC; 1 day 20h ago
 Main PID: 7845 (uwsgi)
    Tasks: 45 (limit: 4915)
   Memory: 569.2M (limit: 32.0G)
      CPU: 5h 43min 35.436s
   CGroup: /system.slice/galaxy.service
           ├─7845 /srv/galaxy/venv/bin/python3 /srv/galaxy/venv/bin/uwsgi --yaml /srv/galaxy/config/galaxy.yml --stats 127.0.0.1:4010
           ├─8106 /srv/galaxy/venv/bin/python3 /srv/galaxy/venv/bin/uwsgi --yaml /srv/galaxy/config/galaxy.yml --stats 127.0.0.1:4010
           ├─8111 /srv/galaxy/venv/bin/python3 /srv/galaxy/venv/bin/uwsgi --yaml /srv/galaxy/config/galaxy.yml --stats 127.0.0.1:4010
           └─8112 /srv/galaxy/venv/bin/python3 /srv/galaxy/venv/bin/uwsgi --yaml /srv/galaxy/config/galaxy.yml --stats 127.0.0.1:4010
```

???

- Here is an example output of the status command.
- The word enabled in the loaded line shows that the service will start at boot
- The service is running since 1 day and 20 hours.
- We are told the main PID.
- And shown the memory usage next to the memory usage limit.
- systemd starts services in a cgroup, so all children can be kept track of.
- Child processes which are started by the parent process are likewise listed.

---

# Supervisor

The former recommended method for managing multiple Galaxy processes

???

- Before systemd there was supervisor.
- We do not use this as much anymore.

---
# Supervisor

A process manager written in Python

- `supervisord` daemon
  - Privileged or unprivileged
- `supervisorctl` command line interface
- INI config format
- `[program:x]` defines a program to control

???

- Supervisor was a process manager written in python.
- It could run as an unprivileged user, and was similar to systemd.

---
class: smaller
# Supervisor - supervisorctl

```console
$ supervisorctl help

default commands (type help <topic>):
=====================================
add    clear  fg        open  quit    remove  restart   start   stop  update
avail  exit   maintail  pid   reload  reread  shutdown  status  tail  version

$ supervisorctl status
galaxy_main:handler0             RUNNING    pid 30554, uptime 7 days, 23:15:11
galaxy_main:handler1             RUNNING    pid 30555, uptime 7 days, 23:15:11
galaxy_main:handler2             RUNNING    pid 30556, uptime 7 days, 23:15:10
galaxy_main:impersonate          RUNNING    pid 30567, uptime 7 days, 23:15:08
galaxy_main:installer            RUNNING    pid 30574, uptime 7 days, 23:15:07
galaxy_main:reports              RUNNING    pid 30563, uptime 7 days, 23:15:09
galaxy_main_supervisord          RUNNING    pid 2108, uptime 8 days, 6:43:54
galaxy_main_uwsgi                RUNNING    pid 3568, uptime 8 days, 6:39:07
nginx                            RUNNING    pid 1917, uptime 8 days, 6:44:21
proftpd                          RUNNING    pid 1916, uptime 8 days, 6:44:21
```

???

- The supervisor status command would show running parent processes.
- Very similar to systemd

---
# Supervisor - program

A config for running a Galaxy server under uWSGI has been installed at `/etc/supervisor/conf.d/galaxy.conf`:

```ini
[program:galaxy]
command         = uwsgi --plugin python --virtualenv /srv/galaxy/server/.venv --ini-paste /srv/galaxy/server/galaxy.yml
directory       = /srv/galaxy/server
autostart       = true
autorestart     = true
startsecs       = 10
user            = galaxy
stopsignal      = INT
```

???

- The service definition for a supervisor process group.
- Here it is named galaxy.
- A command and working directory are provided.
- A user to run the command is listed.
- comparable to systemd service definitions.

---
# Supervisor + systemd

If you prefer, you can use Supervisor to manage the Galaxy processes. The OS needs to ensure the Supervisor daemon is running, and probably manages it with systemd, as is common on most systems currently.

```console
$ systemctl status supervisor
● supervisor.service - Supervisor process control system for UNIX
   Loaded: loaded (/lib/systemd/system/supervisor.service; enabled)
   Active: active (running) since Mon 2016-10-10 14:19:34 EDT; 3 weeks 3 days ago
 Main PID: 481 (supervisord)
   CGroup: /system.slice/supervisor.service
           ├─  481 /usr/bin/python /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf
           ├─ 6264 /usr/bin/uwsgi --plugin python --virtualenv /srv/galaxy/server/.venv --ini-paste /srv/galaxy/server/galaxy.yml
           ├─ 6268 /usr/bin/uwsgi --plugin python --virtualenv /srv/galaxy/server/.venv --ini-paste /srv/galaxy/server/galaxy.yml
           ├─10109 /srv/galaxy/server/.venv/bin/python ./scripts/galaxy-main -c /srv/galaxy/server/galaxy.yml --server-name=handler0
           └─10142 /srv/galaxy/server/.venv/bin/python ./scripts/galaxy-main -c /srv/galaxy/server/galaxy.yml --server-name=handler1
```

???

- These two can be combined.
- systemd can start supervisor, if you have reason to need supervisor.

---

## Thank You!

This material is the result of a collaborative work. Thanks to the [Galaxy Training Network](https://training.galaxyproject.org) and all the contributors!

</div>

</div>

<a rel="license" href="https://creativecommons.org/licenses/by/4.0/">
This material is licensed under the Creative Commons Attribution 4.0 International License</a>.