Reference Genomes in Galaxy

name: inverse
layout: true
class: center, middle, inverse

</span></div>

</span></div>

---

# Reference Genomes in Galaxy

<div class="contributors-line">
		Authors: 
<a href="/training-material/hall-of-fame/blankenberg/" class="contributor-badge contributor-blankenberg"><img src="/training-material/assets/images/orcid.png" alt="orcid logo"/><img src="https://avatars.githubusercontent.com/blankenberg?s=27" alt="Avatar">Daniel Blankenberg</a>

<a href="/training-material/hall-of-fame/slugger70/" class="contributor-badge contributor-slugger70"><img src="https://avatars.githubusercontent.com/slugger70?s=27" alt="Avatar">Simon Gladman</a>

</div>

<div class="footnote" style="bottom: 8em;"><i class="far fa-calendar" aria-hidden="true"></i><span class="visually-hidden">last_modification</span> Updated: Mar 1, 2022</div>

<div class="footnote" style="bottom: 6em;">

<i class="fas fa-file-alt" aria-hidden="true"></i><span class="visually-hidden">text-document</span><a href="slides-plain.html"> Plain-text slides</a>
</div>

<div class="footnote" style="bottom: 2em;">
    <strong>Tip: </strong>press <kbd>P</kbd> to view the presenter notes
    | <i class="fa fa-arrows" aria-hidden="true"></i><span class="visually-hidden">arrow-keys</span> Use arrow keys to move between slides

</div>

???
Presenter notes contain extra information which might be useful if you intend to use these slides for teaching.

Press `P` again to switch presenter notes off

Press `C` to create a new window where the same presentation will be displayed.
This window is linked to the main window. Changing slides on one will cause the
slide to change on the other.

Useful when presenting.

---

# Overview

.large[
* **Intro to built in datasets**
* Built in data hierarchy
* Some problems
* Data Managers
* There's just so much of it!
]

---

# Built in Data

![List_of_data.png](../../images/i06-List_of_data.png)

---

# Data, what data?

.large[
* Some genomes are large! Human, Mouse, Coral
* Some tools require indices of the genomes.
* The indices take a long time to build!
* Better to pre-build the indices.
]

---

# Overview

.large[
* Intro to built in datasets
* **Built in data hierarchy**
* Some problems
* Data Managers
* There's just so much of it!
]

---

# Data schematics in Galaxy

![schematic](../../images/data_managers_schematic_overview.png)

---

# Using reference data in a tool

#### bwa.xml

```xml
<conditional name="reference_source">
      <param name="reference_source_selector" type="select" label="Will you select a reference genome from your history or use a built-in index?" help="Built-ins were indexed using default options. See 'Indexes' section of help below">
        <option value="cached">Use a built-in genome index</option>
        <option value="history">Use a genome from history and build index</option>
      </param>
      <when value="cached">
        <param name="ref_file" type="select" label="Using reference genome" help="Select genome from the list">
          <options from_data_table="bwa_mem_indexes">
            <filter type="sort_by" column="2" />
            <validator type="no_options" message="No indexes are available" />
          </options>
          <validator type="no_options" message="A built-in reference genome is not available for the build associated with the selected input file"/>
        </param>
      </when>
      <when value="history">

```

---

# Where are the data tables?

#### tool_data_table_conf.xml

(Usually located in `galaxy/config/`)

```xml
  <tables>
    
    <table name="bwa_mem_indexes" comment_char="#" allow_duplicate_entries="False">
      <columns>value, dbkey, name, path</columns>
      <file path="tool-data/bwa_index.loc" />
    </table>
  </tables>
```

---

# "loc" files - Short for location!

bwa_index.loc

```text
#
#<unique_build_id>   <dbkey>   <display_name>   <file_path>
#
bosTau7 bosTau7 Cow (bosTau7)   /genomes/bosTau7/bwa_mem_index/bosTau7/bosTau7.fa
ce10    ce10    C. elegans (ce10)       /genomes/ce10/bwa_mem_index/ce10/ce10.fa
danRer7 danRer7 Zebrafish (danRer7)     /genomes/danRer7/bwa_mem_index/danRer7/danRer7.fa
dm3     dm3     D. melanogaster Apr. 2006 (BDGP R5/dm3) (dm3)   /genomes/dm3/bwa_mem_index/dm3/dm3.fa
hg19    hg19    Human (hg19)    /genomes/hg19/bwa_mem_index/hg19/hg19.fa
hg38    hg38    Human (hg38)    /genomes/hg38/bwa_mem_index/hg38/hg38.fa
mm10    mm10    Mouse (mm10)    /genomes/mm10/bwa_mem_index/mm10/mm10.fa
```

---

# Overview

.large[
* Intro to built in datasets
* Built in data hierarchy
* **Some problems**
* Data Managers
* There's just so much of it!
]

---

# Some Problems!

.large[
* Time consuming!
  * ~30 minutes work just to add a new genome to 1 tool!

* Administrator needs to know:
  * how to index **every** tool
  * expected format of the reference data
  * format of the .loc file
]

---

# Typical conversation

.middle[![ref-problem-1.png](../../images/Ref-problem-1.png)]

---

# Typical conversation

.middle[![ref-problem-2.png](../../images/Ref-problem-2.png)]

---

# Typical conversation

.middle[![ref-problem-3.png](../../images/Ref-problem-3.png)]

---

# Typical conversation

.middle[![ref-problem-4.png](../../images/Ref-problem-4.png)]

---

# Other concerns

.large[
* **Accessible?**
  * Manually download genome FASTA files
  * Download, compile, run bwa index; which options?
* **Reproducible?**
  * Only if the person performing manual steps keeps good notes
* **Transparent?**
  * Send email to sysadmin asking for notes
  * Restart Galaxy server for new entries
]

---

# Overview

.large[
* Intro to built in datasets
* Built in data hierarchy
* Some problems
* **Data Managers**
* There's just so much of it!
]

---

# Data Managers

.large[
* Allows for the **creation of built-in** (reference) data
	* underlying data
	* data tables
	* \*.loc files

* Specialized Galaxy tools that can only be accessed by an admin

* Defined **locally** or installed from **ToolShed**
]

---

# Data Managers

.large[
* **Flexible** framework
  * Not just genomic data
  * Run Data Managers through UI
  * Workflow compatible
  * API
* Examples
  * Adding new genome builds (dbkeys)
  * Fetching genome (fasta) sequences
  * Building short read mapper indices for genomes
]

---

# Special class of Galaxy tool

Looks just like a normal Galaxy tool!

![Data-manager-ui.png](../../images/Data-manager-ui.png)

---

# What does it do?

The output of the data manager is a JSON description of the new data table entry

![data_table_JSON.png](../../images/data_table_JSON.png)

This gets turned into a new data table entry

![data_table_entry.png](../../images/data_table_entry.png)

The index files themselves get placed in the appropriate location.

---

# Data Managers Admin

.large[
* Located on the Galaxy's Admin Tab under **Local Data**
]
![data_managers_tool_list.png](../../images/data_managers_tool_list.png)

---

# Data Managers Admin

.large[
* UI tools to fetch reference genomes/build indices
* View progress of index build jobs
* View contents of tool data tables
]
![data_table_ui.png](../../images/data_table_ui.png)

---

# Resources / further reading

.large[
* Galaxy Wiki Page on Data Managers
  * Details
  * Building
  * Examples

https://galaxyproject.org/admin/tools/data-managers/
]

---

# Exercise Time!

---

# Overview

.large[
* Intro to built in datasets
* Built in data hierarchy
* Some problems
* Data Managers
* **There's just so much of it!**
]

---

# There's a lot of reference data

.large[
(and it's hard to keep up with)
]
![ref_data_prob_flow.png](../../images/ref_data_prob_flow.png)

---

# CernVM-FS to the rescue

* Needed a method of sharing reference data across country efficiently
* **CVMFS** is an efficient method for read only data sharing between systems
    * Originally designed for distributed software installation at Cern
    * Turns out it's really useful for read only data sets as well
    * HTTP-based, firewall friendly
* All nodes of Galaxy Main get their reference genomes and indices from CVMFS
    * Shared via mirroring and caching across the country
* It's also really useful to share data **globally**
    * The **usegalaxy.*** initiative has taken full advantage of this.

---

.widen_image[
![cvmfs_server_distribution.png](../../images/cvmfs_server_distribution.png)
]

---

# CVM-FS Global Structure

.widen_image[
![cvmfs_global_structure.png](../../images/cvmfs_global_structure.png)
]

---

# Exercise #2:

.large[
**Connect our instances to CVMFS for reference data**
]

---

## Thank You!

This material is the result of a collaborative work. Thanks to the [Galaxy Training Network](https://training.galaxyproject.org) and all the contributors!

<a href="/training-material/hall-of-fame/slugger70/" class="contributor-badge contributor-slugger70"><img src="https://avatars.githubusercontent.com/slugger70?s=27" alt="Avatar">Simon Gladman</a>

</div>

</div>

<a rel="license" href="https://creativecommons.org/licenses/by/4.0/">
This material is licensed under the Creative Commons Attribution 4.0 International License</a>.