Docker and Galaxy

Contributors

Authors:

Bérénice Batut

Björn Grüning

Questions

Why Docker? What is it?
How to use Docker?
How to integrate Galaxy in Docker to facilitate its deployment?

Objectives

Docker basics
Galaxy Docker image (usage)
Galaxy Docker (internals)
Galaxy flavours

last_modification Last modification: Oct 18, 2022

Preparing Admin Training VMs

If you’re falling along on Galaxy Admin Training VMs, you will need to setup Docker.

You can modify your Ansible setup to configure this as described at bit.ly/gxadmin-docker-playbook.

You will also need to stop Galaxy running on the VM so that you can run Galaxy in a container.

sudo supervisorctl stop galaxy

Speaker Notes

TODO: Remove this slide after the training, it isn’t general.

Preparation

$ docker pull bgruening/galaxy-stable:18.05
$ git clone https://github.com/bgruening/docker-galaxy-stable.git

Speaker Notes

Here you will learn the internals of the Docker Galaxy Image. We will show you tips and tricks on how to run the Galaxy Docker Image successfully in production, how to manage updates and how to bind the container to a cluster scheduler. Moreover, you will learn how to create your own Galaxy flavour mixing a variety of different tools and visualisations

Why Docker?
What is it?

Image of a blue whale with containers on its back.

Deployment issues of bioinformatics tools

Different environments
Different OS
Different packaging
Conflict between tools or versions

Impact on usability and reproducibility

Deployment issue

Galaxy is a box with many dependencies like node, java, nginx, python, uwsgi, anaconda. On the right knime is another box with java and eclipse dependencies. Lines show the multiplicity of tools and dependencies trying to run on multiple environments with windows/osx/linux desktops, servers, and the cloud.

Deployment issue

Matrix from Hell

Galaxy and knime form the y axis of a table with a lot of question marks about which environments support them.

Transport Pre 1960

Icons for multiple types of goods shown multiple methods for transport and storing.

Transport Pre 1960

Matrix from Hell

A similar table full of question marks between goods and shipment methods is shown.

Intermodal shipping container

A blue shipping container is shown solving the transport problem by being a one size container to fit all sorts of goods, on all sorts of transports.

Speaker Notes

A standard container

Virtually loading any goods
Sealed until it reaches final delivery
Can be
- loaded and unloaded
- stacked
- transported efficiently over long distances
- transferred from one mode of transport to another

Docker

Docker logo on a blue shipping container solves the galaxy/knime deployment problem across environments.

Speaker Notes

Docker = An engine that encapsulated any tools as a

lightweight
portable
self sufficient container
manipulated using standard operations
run consistently on virtually any hardware platform

A Docker container?

A schematic with the kernal below and two containers is shown, each container has different stacks.

Lightweight and Open

Speaker Notes

lightweight: same OS kernel, instant start, less RAM use
open: open standard, run on all major unix distributions and windows

Virtual machines vs Containers

Containers more portable and efficient

Speaker Notes

Similar resource isolation and allocation benefits

but a different architectural approach

VM: application + necessary binaries and libraries + entire guest operating system
- All of which can amount to tens of GBs
Container: application and all of its dependencies
- Share the kernel with other containers
- Running as isolated processes in user space on the host operating system

How to use Docker?

A cartoon of a whale with shipping containers atop it.

The client

A command line is explained, docker is the program you're using, run is the subcommand that creates and runs a container, and hello-world is the image you want to load.

Speaker Notes

Docker Engine = the core Docker technology on your computer

Client: to talk with Docker

docker --help

Containers? Images?

A single container labelled galaxy with postgres, nginx, and python inside is shown pointing to three other galaxy containers.

Speaker Notes

Image:

recipe that tell how to build your container
A filesystem and parameters to use at runtime
snapshot of the content of a container: it doesn’t have state and never changes once built

Container

a running instance of an image
you can execute several time the same recipe and if you follow every time the same recipe, you will have the same meal several time. Same for containers

How to get images?

Images are stored in an image registry which are downloaded with docker pull onto a docker host, where you can run multiple instances of galaxy.

Speaker Notes

Docker Registry:

stateless
highly scalable server side application

to stores and distribute Docker images

Example: docker hub, quay.io

`docker pull`

$ docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
c04b14da8d14: Pull complete
Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Status: Downloaded newer image for hello-world:latest

Creation of containers

Images are stored in an image registry which are downloaded with docker pull onto a docker host, where you can run multiple instances of galaxy.

`docker run`

$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker Hub account:
 https://hub.docker.com

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

`docker run`

$ docker run docker/whalesay cowsay Galaxy
Unable to find image 'docker/whalesay:latest' locally
latest: Pulling from docker/whalesay
e190868d63f8: Pull complete
909cd34c6fd7: Pull complete
0b9bfabab7c1: Pull complete
a3ed95caeb02: Pull complete
00bf65475aba: Pull complete
c57b6bcc83e3: Pull complete
8978f6879e2f: Pull complete
8eed3712d2cf: Pull complete
Digest: sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b
Status: Downloaded newer image for docker/whalesay:latest
 ________
< Galaxy >
 --------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/

Speaker Notes

Automatic pull if image not findable
Interaction with the container to say something

`docker run`

$ docker run --help

Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container

  -a, --attach=[]                 Attach to STDIN, STDOUT or STDERR
  --add-host=[]                   Add a custom host-to-IP mapping (host:ip)
  --cpu-shares                    CPU shares (relative weight)
  ...
  -d, --detach                    Run container in background and print container ID
  -e, --env=[]                    Set environment variables
  --entrypoint                    Overwrite the default ENTRYPOINT of the image
  --env-file=[]                   Read in a file of environment variables
  ...
  -h, --hostname                  Container host name
  -i, --interactive               Keep STDIN open even if not attached
  --name                          Assign a name to the container
  --net=default                   Connect a container to a network
  ...
  -P, --publish-all               Publish all exposed ports to random ports
  -p, --publish=[]                Publish a container's port(s) to the host
  --privileged                    Give extended privileges to this container
  --rm                            Automatically remove the container when it exits
  -t, --tty                       Allocate a pseudo-TTY
  -v, --volume=[]                 Bind mount a volume
  ...

Run an interactive container

$ docker run -t -i docker/whalesay
root@7de97f8dd5eb:/cowsay#
root@7de97f8dd5eb:/cowsay# cowsay Galaxy
 ________
< Galaxy >
 --------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/
root@7de97f8dd5eb:/cowsay#

Control during startup

$ docker run -i -t
    -p 8080:80 -p 8021:21 -p 9002:9002
    --privileged=true
    -e "NONUSE=reports"
    -e "GALAXY_CONFIG_ADMIN_USERS=albert@einstein.gov"
    -e "GALAXY_CONFIG_MASTER_API_KEY=83D4jaba7330aDKHkakjGa937"
    -e "GALAXY_CONFIG_BRAND='My own Galaxy flavour'"
    -e "GALAXY_LOGGING=full"

    bgruening/galaxy-stable:18.05

Speaker Notes

possibility to add many different parameters to control the container when starting it

Management of data

$ mkdir data
$ docker run docker/whalesay cowsay Galaxy > data/cowsay
$ more data/cowsay
 ________
< Galaxy >
 --------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/
$ docker run -t -i docker/whalesay
root@f4fa8ed32ef8:/cowsay# ls
ChangeLog  INSTALL  LICENSE  MANIFEST  README  Wrap.pm.diff  cows  cowsay  cowsay.1  install.pl  install.sh  pgp_public_key.txt
root@f4fa8ed32ef8:/cowsay# cowsay Hello Galaxy > cowsay2

Can we access the cowsay file inside the container?
And the cowsay2 file outside the container?

Management of data

A container is shown with an arrow pointing to data and a big x over it.

Speaker Notes

A container is closed

Management of data

The same as before, but inside the container it wants to create a new file in that same folder but it cannot.

Speaker Notes

A container is closed

Data volume

A data volume with two files is shown as a folder, now this is moutned inside two containers.

Speaker Notes

Data volume

a specially-designated directory
within one or more containers
bypasses the Union File System

Data volumes designed to persist data, independent of the container’s life cycle:

Initialisation when a container is created
Possibility of sharing and reusing among containers
Direct changes to a data volume
Persistence of data volumes even if the container itself is deleted

Data volume

Another command line explained, now docker run -v host_dir:container_dir ..., where host_dir is the absolute path or data volume name, and container_dir is where it should be mounted.

Speaker Notes

How to get a data volume?

Data volume

$ ls data/
cowsay_Galaxy
$ docker run -t -i -v $PWD/data:/data docker/whalesay
root@f4fa8ed32ef8:/cowsay# ls /data
cowsay_Galaxy
root@f4fa8ed32ef8:/cowsay# cowsay Galaxy2 > /data/cowsay_Galaxy2
root@f4fa8ed32ef8:/cowsay# ls /data
cowsay_Galaxy  cowsay_Galaxy2
root@f4fa8ed32ef8:/cowsay# exit
$ ls data/
cowsay_Galaxy   cowsay_Galaxy2

Execution of commands
inside a running container

The schematic of a docker host with three galaxies is running, docker exec points to one of the galaxy containers.

Speaker Notes

Run a command in a running container

`docker exec`

$ docker run -d docker/whalesay /bin/sh -c "while true; do sleep 1; done"
7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177
$ docker exec 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 \
cowsay Galaxy
 ________
< Galaxy >
 --------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/

Stop/Start containers

Again the same graphic, but now one of the galaxy containers becomes stopped through the docker stop command and turns grey. Docker start brings it back to running.

Note: Sending SIGTERM and then SIGKILL after a grace period

`docker stop` & `docker start`

$ docker stop 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177
7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177
$
$ docker exec 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 \
cowsay Galaxy
Error response from daemon: Container 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 \
is not running
$
$ docker start 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177
7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177
$
$ docker exec 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 \
cowsay Galaxy
 ________
< Galaxy >
 --------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/

View all containers

docker ps shows a box around the previous graphic and shows running and stopped container.

`docker ps`

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
7179e85085ef        docker/whalesay     "/bin/sh -c 'while tr"   12 minutes ago      Up 2 seconds                            agitated_lovelace
$
$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                        PORTS               NAMES
7de97f8dd5eb        docker/whalesay        "/bin/bash"              4 minutes ago       Exited (0) About a minute ago                       trusting_swanson
9218bbee9c48        docker/whalesay        "/bin/bash"              5 minutes ago       Exited (0) 4 minutes ago                            evil_swirles
7179e85085ef        docker/whalesay        "/bin/sh -c 'while tr"   13 minutes ago      Up 55 seconds                                       agitated_lovelace
ad275579c454        ubuntu                 "/bin/sh -c 'while tr"   15 minutes ago      Exited (137) 13 minutes ago                         condescending_mestorf
66179c4d16da        ubuntu                 "/bin/bash"              About an hour ago   Exited (130) 15 minutes ago                         determined_pasteur
27386c8b69b3        ubuntu                 "/bin/sh"                About an hour ago   Exited (0) About an hour ago                        lonely_ramanujan
4cfefa19e6fa        docker/whalesay        "/bin/bash"              About an hour ago   Exited (0) About an hour ago                        thirsty_chandrasekhar
82687eb94ab9        docker/whalesay        "cowsay Galaxy"          2 hours ago         Exited (0) 2 hours ago                              fervent_babbage
6dbabb9384ad        tmp-bioconda-builder   "/usr/local/bin/tini "   8 days ago          Exited (0) 7 days ago                               tender_bhaskara
5d6f09b94727        tmp-bioconda-builder   "/usr/local/bin/tini "   8 days ago          Exited (0) 8 days ago                               jolly_brattain
4e6f38b4c34c        tmp-bioconda-builder   "/usr/local/bin/tini "   8 days ago          Created                                             angry_colden
b3e6c7412a75        tmp-bioconda-builder   "/usr/local/bin/tini "   8 days ago          Created                                             desperate_visvesvaraya
1ec56c9e37f8        tmp-bioconda-builder   "/usr/local/bin/tini "   8 days ago          Created                                             hopeful_khorana
2b129d00eb10        tmp-bioconda-builder   "/usr/local/bin/tini "   8 days ago          Created                                             gigantic_ptolemy
da45ab698f58        fb77c13d04c0           "/usr/local/bin/tini "   13 days ago         Exited (0) 13 days ago                              jovial_yalow
48dc3ed4e173        fb77c13d04c0           "/usr/local/bin/tini "   13 days ago         Created                                             focused_ritchie
e9195b6512dd        a2107450fdf2           "/usr/local/bin/tini "   2 weeks ago         Created                                             thirsty_bardeen

Note: Name of containers and id

Creation of a new image

A dockerfile built with docker build builds a new container on the local docker host.

Speaker Notes

Need a Dockerfile. I will describe it later. Here it is just to give you a broad overview of the principles

View all images

Docker image shows a list of all the images on the docker host that can be run.

Push your image on a registry

Docker push is shown pushing the locally built container to the image registry.

Hands on! (1)

Play with quay.io/biocontainers/samtools:1.3--1

Get the image
Launch samtools container interactively
Print the help page for samtools
Launch a samtools container in detach mode
Check if it exists and find its name
Stop it and restart it
Print the help page using this container

Hands on! (2)

Play with quay.io/biocontainers/samtools:1.3--1

Create a local samtool_dir directory
Launch an interactive container with a volume pointing at the samtool_dir directory
Add a toy sam file into the local samtool_dir directory
Check if the existence of the file from inside the container
Visualize the content of the toy file with samtools view commands
Generate stats of the toy file into a toy_stat file into the local samtool_dir directory

Speaker Notes

data volume is the way to make your tool connecting

How to containerize your tools?

Cartoon of a docker container

Speaker Notes

Now I am sure all of you want to containerize his tool :)

How to create your image?

Cartoon showing a dockerfile building a container with docker build.

A Dockerfile?

A text document that contains all the commands a user could call on the command line to assemble an image

# Comment
INSTRUCTION arguments

.footnote[Best practices]

Speaker Notes

The paper describing the recipe for you container

Review of the commands

Dockerfile & Layers

dockerfile with steps like run and add and set user and command are shown inverted as the layers of an image, each building on the last.

But Don’t Do it Manually, Use Biocontainers

Tool Dependencies and Containers

How to integrate Galaxy inside Docker Container?

Galaxy Docker Container

A Docker launching a Galaxy instance and

FTP-Server
Webserver
Scheduler
Process control UI
ToolShed ready
Interactive Environment ready

docker run -i -t -p 8080:80 bgruening/galaxy-stable:18.05

Visit your machine on port 8080 to see Galaxy running.

https://github.com/bgruening/docker-galaxy-stable

Hands on!

Launch a Galaxy Docker Container and try to

Add Data
Become an Admin
Restart Galaxy
Use export mounts (persistent data)
Trouble shooting (logging)
Install tools
…

Galaxy + SLURM =

https://github.com/bgruening/docker-galaxy-stable/tree/dev/docs/Running_jobs_outside_of_the_container.md

Docker flavours

Schematic of multiple docker containers and how they layer upon one another. ubuntu forms the base, and requirements builds on that, then galaxy-stable on top of that.

Galactic Flavours

Create your own Flavour

# Galaxy - My own flavour
#
# VERSION       0.1

FROM bgruening/galaxy-stable:18.05

MAINTAINER Björn A. Grüning, bjoern.gruening@gmail.com

ENV GALAXY_CONFIG_BRAND deepTools

# Adding the tool definitions to the container
ADD my_tool_list.yml $GALAXY_ROOT/my_tool_list.yml

# Install deepTools
RUN install-tools $GALAXY_ROOT/my_tool_list.yml

The tools file

galaxy_instance: http://localhost:8080
tools:
- name: fastqc
  owner: devteam
  tool_panel_section_id: cshl_library_information
  tool_shed_url: https://toolshed.g2.bx.psu.edu
  #revisions:

Hands on!

Create a Galaxy Flavour

Galaxy Flavor Generator

Website used to build custom flavours of galaxy docker containers

https://github.com/galaxyFlavorsGenerators/galaxyFlavorGenerator

Key Points

Use Docker
Integrate your tools into Galaxy
Use Galaxy inside Docker to have a flavoured Galaxy instance

Thank you!

This material is the result of a collaborative work. Thanks to the Galaxy Training Network and all the contributors! page logo

This material is licensed under the Creative Commons Attribution 4.0 International License.

Docker and Galaxy

Contributors

Questions

Objectives

Preparing Admin Training VMs

Preparation

Why Docker? What is it?

Deployment issues of bioinformatics tools

Deployment issue

Deployment issue

Transport Pre 1960

Transport Pre 1960

Intermodal shipping container

Docker

A Docker container?

Virtual machines vs Containers

How to use Docker?

The client

Containers? Images?

How to get images?

docker pull

Creation of containers

docker run

docker run

docker run

Run an interactive container

Control during startup

Management of data

Management of data

Management of data

Data volume

Data volume

Data volume

Execution of commands inside a running container

docker exec

Stop/Start containers

docker stop & docker start

View all containers

docker ps

Creation of a new image

View all images

Push your image on a registry

How to containerize your tools?

How to create your image?

A Dockerfile?

Dockerfile & Layers

But Don’t Do it Manually, Use Biocontainers

How to integrate Galaxy inside Docker Container?

Galaxy Docker Container

Galaxy + SLURM =

Docker flavours

Galactic Flavours

Create your own Flavour

The tools file

Galaxy Flavor Generator

Key Points

Thank you!

Why Docker?
What is it?

`docker pull`

`docker run`

`docker run`

`docker run`

Execution of commands
inside a running container

`docker exec`

`docker stop` & `docker start`

`docker ps`