name: inverse layout: true class: center, middle, inverse <div class="my-header"><span> <a href="/training-material/topics/dev" title="Return to topic page" ><i class="fa fa-level-up" aria-hidden="true"></i></a> <a class="nav-link" href="https://github.com/galaxyproject/training-material/edit/main/topics/dev/tutorials/toolshed/slides.html"><i class="fa fa-pencil" aria-hidden="true"></i></a> </span></div> <div class="my-footer"><span> <img src="/training-material/assets/images/GTN-60px.png" alt="Galaxy Training Network" style="height: 40px;"/> </span></div> --- <img src="/training-material/assets/images/GTN.png" alt="Galaxy Training Network" class="cover-logo"/> # Tool Shed: sharing Galaxy tools <div markdown="0"> <div class="contributors-line"> Authors: <a href="/training-material/hall-of-fame/abretaud/" class="contributor-badge contributor-abretaud"><img src="/training-material/assets/images/orcid.png" alt="orcid logo"/><img src="https://avatars.githubusercontent.com/abretaud?s=27" alt="Avatar">Anthony Bretaudeau</a> <a href="/training-material/hall-of-fame/loraine-gueguen/" class="contributor-badge contributor-loraine-gueguen"><img src="https://avatars.githubusercontent.com/loraine-gueguen?s=27" alt="Avatar">Loraine Guéguen</a> <a href="/training-material/hall-of-fame/slegras/" class="contributor-badge contributor-slegras"><img src="https://avatars.githubusercontent.com/slegras?s=27" alt="Avatar">Stéphanie Legras</a> <a href="/training-material/hall-of-fame/bebatut/" class="contributor-badge contributor-bebatut"><img src="/training-material/assets/images/orcid.png" alt="orcid logo"/><img src="https://avatars.githubusercontent.com/bebatut?s=27" alt="Avatar">Bérénice Batut</a> </div> </div> <!-- modified date --> <div class="footnote" style="bottom: 8em;"><i class="far fa-calendar" aria-hidden="true"></i><span class="visually-hidden">last_modification</span> Updated: Sep 28, 2022</div> <!-- other slide formats (video and plain-text) --> <div class="footnote" style="bottom: 6em;"> <i class="fas fa-file-alt" aria-hidden="true"></i><span class="visually-hidden">text-document</span><a href="slides-plain.html"> Plain-text slides</a> </div> <!-- usage tips --> <div class="footnote" style="bottom: 2em;"> <strong>Tip: </strong>press <kbd>P</kbd> to view the presenter notes | <i class="fa fa-arrows" aria-hidden="true"></i><span class="visually-hidden">arrow-keys</span> Use arrow keys to move between slides </div> ??? Presenter notes contain extra information which might be useful if you intend to use these slides for teaching. Press `P` again to switch presenter notes off Press `C` to create a new window where the same presentation will be displayed. This window is linked to the main window. Changing slides on one will cause the slide to change on the other. Useful when presenting. --- ### <i class="far fa-question-circle" aria-hidden="true"></i><span class="visually-hidden">question</span> Questions - What is a Tool Shed? - How to install tools and workflows from a Tool Shed into a Galaxy instance? - What are the Tool Shed repository types? - How to publish with Planemo? --- ### <i class="fas fa-bullseye" aria-hidden="true"></i><span class="visually-hidden">objectives</span> Objectives - Discover what is a Tool Shed - Be able to install tools and workflows from a Tool Shed into a Galaxy instance - Be able to publish tools with Planemo --- ## 1. What is a Tool Shed?  --- ###  = Galaxy App Store --- ###  = Galaxy App Store - Galaxy tools - ~~Recipes for installing and compiling tools~~, - Data Managers, - Custom datatypes, - Exported workflows. --- ###  = Galaxy App Store - For Galaxy administrators, facilitates: - **installing/updating** tools - For tool developers, facilitates: - **sharing** of Galaxy utilities - **versioning** --- ### Available Tool Sheds - [Galaxy community Tool Shed](https://toolshed.g2.bx.psu.edu/) -> Main Tool Shed - [Galaxy community test Tool Shed](https://testtoolshed.g2.bx.psu.edu/) -> Sandbox for testing - [List of public Tool Sheds](https://galaxyproject.org/toolshed/public-toolsheds/) --- ### Tool Shed interface  --- ### Example of tool  --- ## 2. Installing tools from a Tool Shed --- ### Connect your Galaxy to a Tool Shed - Only for third party toolsheds (Main and Test by default) - Add `<tool_shed />` block in `config/tool_sheds_conf.xml` - Restart Galaxy --- ### Install a tool from the Tool Shed - Go to the admin interface and click on "Search Tool Shed"  - Select a Tool Shed  --- ### Install a tool from the Tool Shed - Search your tool  --- ### Install a tool from the Tool Shed - Types of repository</span> - <span style="color:orange"> Tools (`$name`)</span> - <span style="color:blue"> Tool suites (`suite_$name`)</span> - <span style="color:red"> Tool dependencies (`package_$name_$version`)</span> (gradually removed, replaced by Conda dependencies)  --- ### Install a tool from the Tool Shed  - Install the tool  --- ### Install a tool from the Tool Shed - Install the tool  - <span style="font-size:70%"> Selecting Tool Shed AND conda will make Galaxy to install both</span> - <span style="font-size:70%"> Recommended: use conda </span> --- ### Install a tool from the Tool Shed - Check  --- ### Manage installed tools - Admin > Manage installed tools   --- ### Manage installed tools - Click on the name of a tool - Manage and browse the repository  --- ## Tips: How to install workflows --- ### How to install a workflow? - Admin > Search Tool Shed - Search the workflow 1.  2.  - <span style="font-size:70%"> Tools can come with the workflow if they are not yet installed </span> --- ### How to install a workflow?  - No workflow? --- ### How to install a workflow?  - Click on the imported workflow --- ### How to install a workflow?  - Click on the workflow at the bottom of the “Contents of this repository” section --- ### How to install a workflow? - Repository Actions > Import workflow to Galaxy  --- ### How to install a workflow?  - Done! --- ## 3. Tool Shed repository types --- ### 3.1 Tool with dependencies --- ### Tool with dependencies  --- ### 3.2 Tool suite #### Group of related repositories --- ### Tool suite  - Works but no way to install a single tool --- ### Tool suite  - One repository per individual tool - A special 'Tool suite' repository listing the individual repositories --- ### Tool suite - relationship between a dependent repository and one or more required repository (tools, datatypes, workflows,...) - -> One XML file ```xml <?xml version="1.0"?> <repositories description="Pipeline phylogeny"> <repository toolshed="https://testtoolshed.g2.bx.psu.edu" name="fasta_to_phylip" owner="gandres" changeset_revision="a895633568" /> <repository name="mafft" owner="gandres" /> <repository name="phyml" owner="gandres" /> […] </repositories> ``` No need to write it yourself, Planemo does it for you --- ### Tool suite  --- ### 3.3 Workflow --- ### Workflow  - A succession of tools to perform a complete analysis --- ### Workflow - In galaxy  --- ### Workflow  - You get a file: Galaxy-Workflow-NAME.ga --- ### Workflow - You have your file Galaxy-Workflow-NAME.ga - How to automatically install the tools used in the workflow? - -> `repository_dependencies.xml` ```xml <?xml version="1.0"?> <repositories description="Phylogeny workflow"> <repository name="suite_phylogeny" owner="galaxy" /> </repositories> ``` --- ### Workflow  --- ### Workflow  - **Everything must be on the same toolshed** --- ## Tips ### What about tools working with *.loc files or new datatypes --- ### Tools with `*.loc` files or new datatypes - add them to the Tool Shed repository. - `*.loc` files and `datatypes` files should be provided as `*.sample` file in a `tool-data` directory. - `tool-data/*.loc.sample` files come along with `tool_data_table_conf.xml.sample`. Their content or the file themselves are put to the right location on tool set up. --- ### Tools with *.loc files or new datatypes  <span style="font-size:60%">`tool_data_table_conf.xml.sample`</span> ```xml <tables> <!-- Locations of indexes in the Bowtie2 mapper format --> <table name="bowtie2_indexes" comment_char="#"> <columns>value, dbkey, name, path</columns> <file path="tool-data/bowtie2_indices.loc" /> </table> </tables> ``` <span style="font-size:60%">On install, the content of the file and the right path to the *.loc file is added to `shed_tool_data_table_conf.xml`</span> --- ## 4. Publishing with Planemo --- <i class="fas fa-exclamation-triangle" aria-hidden="true"></i><span class="visually-hidden">warning</span> *There are automated ways to publish to a ToolShed using GitHub and Continuous Integration* [Check out these slides, "Publishing tools" section](/training-material/topics/dev/tutorials/tool-integration/slides.html) --- ### Configure the target Tool Shed ```bash planemo config_init ``` - Creates `.planemo.yml` file in your home directory: ```yml ## Planemo Global Configuration File. ## Everything in this file is completely optional - these values can all be ## configured via command line options for the corresponding commands. ## [...] sheds: toolshed: #key: "<TODO>" #email: "<TODO>" #password: "<TODO>" testtoolshed: #key: "<TODO>" #email: "<TODO>" #password: "<TODO>" local: #key: "<TODO>" #email: "<TODO>" #password: "<TODO>" ``` --- ### Configure the target Tool Shed - Add your Tool Shed API key in `.planemo.yml` ```yml ## Planemo Global Configuration File. ## Everything in this file is completely optional - these values can all be ## configured via command line options for the corresponding commands. ## [...] sheds: toolshed: key: "xxxxxxxxxmytoolshedAPIkeyxxxxxxx" #email: "<TODO>" #password: "<TODO>" testtoolshed: #key: "<TODO>" #email: "<TODO>" #password: "<TODO>" local: #key: "<TODO>" #email: "<TODO>" #password: "<TODO>" ``` --- ### Configure repository ```bash planemo shed_init --name="tandem_repeats_finder_2" --owner="gandres" --description="Tandem Repeats Finder description" --long_description="A long long description." --category="Sequence Analysis" [--remote_repository_url=<URL to .shed.yml on github>] [--homepage_url=<Homepage for tool.>] ``` - Creates `.shed.yml` file in the current repository (tool repository): ```yml categories: [Sequence Analysis] description: Tandem Repeats Finder description long_description: A long long description. name: tandem_repeats_finder_2 owner: gandres ``` --- ### Configure repository #### Naming conventions <small> | Repository Type | Recommended Name | Examples | | --------------- |--------------------------| -------------------------| | Data Managers | `data_manager_$name` | `data_manager_bowtie2` | | Packages | `package_$name_$version` | `package_aragorn_1_2_36` | | Tool Suites | `suite_$name` | `suite_samtools` | | Tools | `$name` | `stringle`, `bowtie2` | </small> <span style="color:red">Planemo will determine the repository type based on these conventions</span> --- ### Configure repository - Tools and tool suite `.shed.yml`: implicitly define repositories for each individual tool in the directory and build a suite for those tools. - Example: `trinity/.shed.yml` ```yml [...] auto_tool_repositories: name_template: "" description_template: " (from the Trinity tool suite)" suite: name: "suite_trinity" description: Trinity tools to assemble transcript sequences from Illumina RNA-Seq data. ``` --- ### Check ```bash planemo shed_lint --tools ``` ```bash Linting repository […]/tandem_repeats_finder Applying linter expansion... CHECK .. INFO: Included files all found. Applying linter tool_dependencies_xsd... CHECK .. INFO: tool_dependencies.xml found and appears to be valid XML Applying linter tool_dependencies_actions... CHECK .. INFO: Parsed tool dependencies. Applying linter repository_dependencies... CHECK .. INFO: No repository_dependencies.xml, skipping. Applying linter shed_yaml... CHECK .. INFO: .shed.yml found and appears to be valid YAML. Applying linter readme... CHECK .. INFO: No README found skipping. +Linting tool […]/tandem_repeats_finder/tandem_repeats_finder_wrapper.xml […] ``` --- ### Create repository ```bash planemo shed_create --shed_target testtoolshed ``` - Don't forget **--shed_target**. Default is main toolshed --- ### Create repository ```bash planemo shed_create --shed_target testtoolshed ``` ```bash Repository created cd '[…]/tandem_repeats_finder' && git rev-parse HEAD Repository tandem_repeats_finder_2 updated successfully. ```  --- ### Update repository ```bash planemo shed_diff --shed_target testtoolshed ``` ```bash wget -q --recursive -O - 'https://testtoolshed.g2.bx.psu.edu/repository/download?repository_id=6306166e4da9a8bf&changeset_revision=default&file_type=gz' | tar -xzf - -C /tmp/tool_shed_diff_DMcOrR/_testtoolshed_ --strip-components 1 mkdir "/tmp/tool_shed_diff_DMcOrR/_local_"; tar -xzf "/tmp/tmp5ousfg" -C "/tmp/tool_shed_diff_DMcOrR/_local_"; rm -rf /tmp/tmp5ousfg cd "/tmp/tool_shed_diff_DMcOrR"; diff -r _local_ _testtoolshed_ diff -r _local_/tandem_repeats_finder.xml _testtoolshed_/tandem_repeats_finder.xml 41d40 < <param name="fileout_label2" type="text" value="Unused" label="none" help="Output name for files" /> ``` - The repository needs to be updated --- ### Update repository ```bash planemo shed_update --check_diff --shed_target testtoolshed ``` ```bash wget -q --recursive -O - 'https://testtoolshed.g2.bx.psu.edu/repository/download?repository_id=6306166e4da9a8bf&changeset_revision=default&file_type=gz' | tar -xzf - -C /tmp/tool_shed_diff_DMcOrR/_testtoolshed_ --strip-components 1 mkdir "/tmp/tool_shed_diff_DMcOrR/_local_"; tar -xzf "/tmp/tmp5ousfg" -C "/tmp/tool_shed_diff_DMcOrR/_local_"; rm -rf /tmp/tmp5ousfg cd "/tmp/tool_shed_diff_DMcOrR"; diff -r _local_ _testtoolshed_ diff -r _local_/tandem_repeats_finder.xml _testtoolshed_/tandem_repeats_finder.xml 41d40 < <param name="fileout_label2" type="text" value="Unused" label="none" help="Output name for files" /> Repository tandem_repeats_finder updated successfully. Repository metadata updated. ``` - The repository has been updated --- ### Update repository ```bash planemo shed_update --check_diff --shed_target testtoolshed ```  - The repository has been updated ---  #### Publishing to the Tool Shed  ---  > Command-line utilities to assist in building and publishing Galaxy tools. ---   ---  #### Checklist (3/3) - Initiate the `.shed.yml` file with ```bash planemo shed_init ``` - Create the repository on ToolShed with ```bash planemo shed_create ``` - Install and test the wrapper on a Galaxy instance - Improve the wrapper and push modification on toolshed ```bash planemo shed_update ``` --- ## References - [ToolShed](https://galaxyproject.org/toolshed/tour/) - [Install tools from the ToolShed](https://galaxyproject.org/admin/tools/add-tool-from-toolshed-tutorial/) - [Publish tools with Planemo](https://planemo.readthedocs.io/en/latest/) --- <div id="hands-on-hands-on" class="box-title" aria-label="hands-on box: Hands-on"><i class="fas fa-pencil-alt" aria-hidden="true"></i><span class="visually-hidden"></span> Hands-on: Hands-on</div>  --- ### Hands-on #### The goal Publish your tool (seqtk_seq) in your local toolshed with Planemo --- <div id="hands-on-hands-on-1" class="box-title" aria-label="hands-on box: Hands-on"><i class="fas fa-pencil-alt" aria-hidden="true"></i><span class="visually-hidden"></span> Hands-on: Hands-on</div> #### Step 1: Start and configure your toolshed - Start your toolshed (in galaxy home directory): ```bash ./run_tool_shed.sh ``` - Toolshed available at `http://localhost:9009` <aside class="notes"> login: galaxy/azerty </aside> --- <div id="hands-on-hands-on-2" class="box-title" aria-label="hands-on box: Hands-on"><i class="fas fa-pencil-alt" aria-hidden="true"></i><span class="visually-hidden"></span> Hands-on: Hands-on</div> #### Step 2: Use planemo to publish your tool --- <div id="hands-on-hands-on-3" class="box-title" aria-label="hands-on box: Hands-on"><i class="fas fa-pencil-alt" aria-hidden="true"></i><span class="visually-hidden"></span> Hands-on: Hands-on</div> #### Step 2: Use planemo to publish your tool ```bash planemo config_init ``` - Creates file `.planemo.yml` in your home directory. ```yml [...] local: #key: "<TODO>" email: "galaxy@localhost.fr" password: "azerty" ``` ```bash planemo shed_init --name=seqtk_seq --owner=galaxy --description="common transformation of FASTA/Q" --category="Fasta Manipulation" ``` ```bash planemo shed_lint . ``` ```bash planemo shed_create --shed_target local ``` --- <div id="hands-on-hands-on-4" class="box-title" aria-label="hands-on box: Hands-on"><i class="fas fa-pencil-alt" aria-hidden="true"></i><span class="visually-hidden"></span> Hands-on: Hands-on</div> #### Step 3: Update your tool and then use planemo to update your tool on the Tool Shed --- <div id="hands-on-hands-on-5" class="box-title" aria-label="hands-on box: Hands-on"><i class="fas fa-pencil-alt" aria-hidden="true"></i><span class="visually-hidden"></span> Hands-on: Hands-on</div> #### Step 3: Update your tool and then use planemo to update your tool on the Tool Shed ```bash planemo shed_diff --shed_target local ``` ```bash planemo shed_update --check_diff --shed_target local ``` --- ### <i class="fas fa-key" aria-hidden="true"></i><span class="visually-hidden">keypoints</span> Key points - Use Planemo - Contribute to the Main Tool Shed --- ## Thank You! This material is the result of a collaborative work. Thanks to the [Galaxy Training Network](https://training.galaxyproject.org) and all the contributors! <div markdown="0"> <div class="contributors-line"> Authors: <a href="/training-material/hall-of-fame/abretaud/" class="contributor-badge contributor-abretaud"><img src="/training-material/assets/images/orcid.png" alt="orcid logo"/><img src="https://avatars.githubusercontent.com/abretaud?s=27" alt="Avatar">Anthony Bretaudeau</a> <a href="/training-material/hall-of-fame/loraine-gueguen/" class="contributor-badge contributor-loraine-gueguen"><img src="https://avatars.githubusercontent.com/loraine-gueguen?s=27" alt="Avatar">Loraine Guéguen</a> <a href="/training-material/hall-of-fame/slegras/" class="contributor-badge contributor-slegras"><img src="https://avatars.githubusercontent.com/slegras?s=27" alt="Avatar">Stéphanie Legras</a> <a href="/training-material/hall-of-fame/bebatut/" class="contributor-badge contributor-bebatut"><img src="/training-material/assets/images/orcid.png" alt="orcid logo"/><img src="https://avatars.githubusercontent.com/bebatut?s=27" alt="Avatar">Bérénice Batut</a> </div> </div> <div style="display: flex;flex-direction: row;align-items: center;justify-content: center;"> <img src="/training-material/assets/images/GTN.png" alt="Galaxy Training Network" style="height: 100px;"/> </div> <a rel="license" href="https://creativecommons.org/licenses/by/4.0/"> This material is licensed under the Creative Commons Attribution 4.0 International License</a>.