Using Workflow Parameters

Overview
Questions:
  • What are Workflow Parameters

  • How can I define and use Workflow Parameters

  • How can I read Parameters from Datasets

Objectives:
  • Learn how to use Workflow Parameters to improve your Workflows

Time estimation: 30 minutes
Level: Intermediate Intermediate
Supporting Materials:
Last modification: Oct 18, 2022
License: Tutorial Content is licensed under Creative Commons Attribution 4.0 International License The GTN Framework is licensed under MIT

Introduction

Workflows are a powerful feature in Galaxy that allow you to chain multiple steps of an analysis together. To make a workflow reusable with slightly different settings you can define and use workflow parameters.

Agenda

In this tutorial, we will cover:

  1. Introduction
  2. Add workflow parameters to a workflow
  3. Add an integer workflow parameter
  4. Create a text parameter from multiple parameters
  5. Read a parameter from a dataset
  6. Conclusion

Add workflow parameters to a workflow

We will import a simple workflow and then demonstrate how workflow parameters can be added.

Hands-on: Import integer workflow
  1. Click on Workflow in the top menu
  2. Click on the galaxy-upload import workflow button on the right
  3. Enter the following URL into the “Archived Workflow” URL box

    https://raw.githubusercontent.com/galaxyproject/training-material/master/topics/galaxy-interface/tutorials/workflow-parameters/workflows/cut_n_lines.ga
    
  4. Click on import workflow

Add an integer workflow parameter

This workflow takes an input dataset and outputs the first 10 lines of that input dataset. The number of lines that should be output can be set using the Number of lines parameter of the select first tool.

Instead of selecting a specific number of lines we can choose to insert a workflow parameter that allows the user to set this parameter when running the workflow.

Button for selecting connections.
Figure 1: The new button for creating a workflow parameter connection. When you click this, it will add a new 'input' to the tool that can be connected, just like other tools in workflows.
Hands-on: Updating the Workflow
  1. Upload a dataset using “Paste/Fetch data” with the contents:

    E. coli (O19:H34. Extracellular pathogenic)
    E. coli (O7:K1. Extracellular pathogenic)
    E. coli 536
    E. coli APEC O1 (O1:K12:H7. APEC
    E. coli CFT083 (O6:K2:H1. UPEC)
    E. coli E2348/69 (O127:H6. Enteropathogenic)
    E. coli EC4115 (O157:H7 EHEC)
    E. coli ED1a O81 (Commensal)
    E. coli EDL933 (O157:H7 EHEC)
    E. coli F11
    E. coli S88 (O45:K1. Extracellular pathogenic)
    E. coli SE15 (O150:H5. Commensal)
    E. coli Sakai (O157:H7 EHEC)
    E. coli TW14359 (O157:H7 EHEC)
    E. coli UMN026 (O17:K52:H18. Extracellular pathogenic)
    E. coli UTI89 O18:K1:H7. UPEC)
    
  2. Open the workflow editor

    1. In the top menu bar, click on Workflows
    2. Click on the name of the workflow you want to editWorkflow drop down menu showing Edit option
    3. Select galaxy-wf-edit Edit from the dropdown menu to open the workflow in the workflow editor
  3. Click the Select first tool tool in your workflow
  4. Find the Number of lines parameter in the right hand panel.
  5. Click on the workflow connection button galaxy-wf-connection to convert the Number of lines parameter into a workflow parameter connection

    You will see that a new input with the text Number of lines has appeared on the Select first tool in the editor.

  6. Add a Simple inputs used for workflow logic, found under the Inputs section of your toolbox
  7. If you click on the Input parameter box in the editor you will see in the Details panel that the Parameter type is set to Text. If you try to connect this parameter to the “Number of lines” parameter you will see that the noodle turns orange and that you cannot create a connection.

  8. Click on Input parameter, and change the Parameter type to Integer

  9. Connect “output” from Input parameter tool to the “Number of lines” input of the Select first tool.

  10. Save galaxy-save your workflow

  11. Run your workflow

    • Click on Workflow on the top menu bar of Galaxy. You will see a list of all your workflows.
    • Click on the workflow-run (Run workflow) button next to your workflow
    • Configure the workflow as needed
    • Click the Run Workflow button at the top-right of the screen
    • You may have to refresh your history to see the queued jobs

Notice the new input that can be changed before the workflow is run

Animation of simple integer workflow parameter.
Figure 2: Integer workflow parameter

You can connect a parameter to multiple steps or to multiple parameters within a step, everywhere you need to use it.

Create a text parameter from multiple parameters

It is often necessary to create a text parameter that is flanked by additional text. Take for instance a regular expression that finds foo in a string and replaces it with foobar as in s/(foo)/\1bar/. The Galaxy tool Regex Find And Replace can be used to tune such a regular expression. The find part of the regex (s/(foo)/) can be defined in the Find regex parameter and the replacement part can be entered in the Replacement parameter. If we want to make the foo part of the regular expression configurable we can compose this text parameter using the Compose text parameter value tool.

Hands-on: Compose a text parameter
  1. Create a new workflow

    1. Click Workflow on the top bar
    2. Click the new workflow galaxy-wf-new button
    3. Give it a clear and memorable name
    4. Clicking Save will take you directly into the workflow editor for that workflow
    5. Need more help? Please see the How to make a workflow subsection here
  2. Add an Inputs → Input Dataset to the workflow
  3. Add an Inputs → Simple inputs used for workflow logic to the workflow
    • param-select “Parameter type”: Text
  4. Add Compose text parameter value tool to the workflow
    • Add two more “Components” using the “Insert components” button in the workflow editor tool form on the right of the screen. You will need three “Component” blocks in total.
    • In the first component:
      • “Enter text that should be part of the computed value”: (
    • In the second component:
      • “Enter text that should be part of the computed value”: Leave empty and click “Add connection to module” galaxy-wf-connection
      • Connect the output of the Input parameter tool to this new input
    • In the third component:
      • “Enter text that should be part of the computed value”: )
  5. Add the Regex Find And Replace tool to the workflow
    • Click on Insert Check
    • “Find Regex”: Click “Add connection to module” galaxy-wf-connection
    • “Replacement”: \1bar
    • Connect the output of the Compose text parameter value tool to the “Find Regex” parameter of Regex Find And Replace tool.
    • Connect the output of the Input dataset tool to the “Select lines from” input of the Regex Find And Replace tool.
  6. Save galaxy-save your workflow

You’ve now built a workflow with a parameterised input! It’s time to test it out.

Hands-on: Running the workflow
  1. Upload a dataset using “Paste/Fetch data” with the contents wunder

  2. Run your workflow with the following parameters

    • “Input parameter”: wunder

      This is the value that will be looked for in your input dataset.

    • Click on Workflow on the top menu bar of Galaxy. You will see a list of all your workflows.
    • Click on the workflow-run (Run workflow) button next to your workflow
    • Configure the workflow as needed
    • Click the Run Workflow button at the top-right of the screen
    • You may have to refresh your history to see the queued jobs
  3. Examine the outputs

You should see two new datasets in your history. The first dataset has the data type expression.json and contains the composed parameter value (wunder), the second dataset will contain the output of the Regex Find And Replace tool step. A click on the information details button will show the parameters for the tool. You will see that the “Find Regex” parameter will contain the values that you entered in the workflow run form. If you look at the dataset content you will see it is wunderbar.

Read a parameter from a dataset

Often times it is necessary to calculate a parameter in one step of a workflow and then to use it in another step of the same workflow. This can be accomplished by reading the parameter from a dataset (As long as it is a text, integer, float, boolean or color parameter). In this example we will construct a workflow where we calculate the sum of all values in a dataset and then divide the values in this dataset by the sum calculated in the previous step.

Hands-on: Construct Workflow with Parameters read from a dataset
  1. Create a new workflow
  2. Add an Input dataset tool
  3. Add Datamash (operations on tabular data) tool to the workflow
    • wf-input “Input tabular dataset”: Connect the noodle from the output of the Input dataset tool to this input
    • param-repeat Operation to perform in each group
      • param-select “Type”: sum
      • param-text “On column”: 1
  4. Add Parse parameter value tool to the workflow
    • “Select type of parameter to parse”: integer
    • wf-input “Input file containing parameter to parse out of”: Connect the Datamash tool output to this input
  5. Add Compose text parameter value tool to the workflow
    • In the first repeat:
      • param-select “Choose the type of parameter for this field”: Text Parameter
      • param-text “Enter text that should be part of the computed value”: c1/
    • Click Insert Repeat
    • In the second repeat:
      • param-select “Choose the type of parameter for this field”: Integer Parameter
      • “Enter integer that should be part of the computed value”: Click on Add connection to module galaxy-wf-connection
      • wf-input “Input”: Connect the output of the Parse parameter value tool
  6. Add Compute an expression on every row tool to the workflow
    • “Add expression as a new column to”: click on Add connection to module, then connect the output of Compose text parameter value tool
    • “as a new column to”: Select the output of the Input dataset tool

With this you’re ready to run the workflow!

Hands-on: Running the workflow
  1. Upload a dataset using “Paste/Fetch data” with the contents, and set the filetype manually to “tabular” during upload

    16378
    16014
    2831
    5702
    149
    24383
    12220
    4488
    11500
    24724
    
  2. Run your workflow with the following parameters

    • “Input dataset”: the table you have just uploaded
    • Click on Workflow on the top menu bar of Galaxy. You will see a list of all your workflows.
    • Click on the workflow-run (Run workflow) button next to your workflow
    • Configure the workflow as needed
    • Click the Run Workflow button at the top-right of the screen
    • You may have to refresh your history to see the queued jobs
  3. Examine the outputs

This workflow will produce a new dataset, where the last column will be the result of dividing the value in the first column by the sum of all values in the first column.

  • Often times you need to try a couple of different parameter values and pick the best one. You can create or compute a dataset for each parameter you would like to try and run the Parse parameter value tool on it, running all downstream tools once for each parameter.

Conclusion

Galaxy Workflows chain together different steps of an analysis. To make your workflows more useful to your colleagues you can add workflow parameters. Sometimes a parameter is not known in advance, but can be calculated as part of the workflow. Now you know how to read these parameters from datasets!

Key points
  • Use Workflow Parameters to make your Workflows more versatile

Frequently Asked Questions

Have questions about this tutorial? Check out the tutorial FAQ page or the FAQ page for the Using Galaxy and Managing your Data topic to see if your question is listed there. If not, please ask your question on the GTN Gitter Channel or the Galaxy Help Forum

Feedback

Did you use this material as an instructor? Feel free to give us feedback on how it went.
Did you use this material as a learner or student? Click the form below to leave feedback.

Click here to load Google feedback frame

Citing this Tutorial

  1. Marius van den Beek, Helena Rasche, 2022 Using Workflow Parameters (Galaxy Training Materials). https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/workflow-parameters/tutorial.html Online; accessed TODAY
  2. Batut et al., 2018 Community-Driven Data Analysis Training for Biology Cell Systems 10.1016/j.cels.2018.05.012


@misc{galaxy-interface-workflow-parameters,
author = "Marius van den Beek and Helena Rasche",
title = "Using Workflow Parameters (Galaxy Training Materials)",
year = "2022",
month = "10",
day = "18"
url = "\url{https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/workflow-parameters/tutorial.html}",
note = "[Online; accessed TODAY]"
}
@article{Batut_2018,
    doi = {10.1016/j.cels.2018.05.012},
    url = {https://doi.org/10.1016%2Fj.cels.2018.05.012},
    year = 2018,
    month = {jun},
    publisher = {Elsevier {BV}},
    volume = {6},
    number = {6},
    pages = {752--758.e1},
    author = {B{\'{e}}r{\'{e}}nice Batut and Saskia Hiltemann and Andrea Bagnacani and Dannon Baker and Vivek Bhardwaj and Clemens Blank and Anthony Bretaudeau and Loraine Brillet-Gu{\'{e}}guen and Martin {\v{C}}ech and John Chilton and Dave Clements and Olivia Doppelt-Azeroual and Anika Erxleben and Mallory Ann Freeberg and Simon Gladman and Youri Hoogstrate and Hans-Rudolf Hotz and Torsten Houwaart and Pratik Jagtap and Delphine Larivi{\`{e}}re and Gildas Le Corguill{\'{e}} and Thomas Manke and Fabien Mareuil and Fidel Ram{\'{\i}}rez and Devon Ryan and Florian Christoph Sigloch and Nicola Soranzo and Joachim Wolff and Pavankumar Videm and Markus Wolfien and Aisanjiang Wubuli and Dilmurat Yusuf and James Taylor and Rolf Backofen and Anton Nekrutenko and Björn Grüning},
    title = {Community-Driven Data Analysis Training for Biology},
    journal = {Cell Systems}
}
                   

Congratulations on successfully completing this tutorial!