Merge pull request #1919 from nsoranzo/tool_dependencies

Re-add Tool Dependencies slides to admin

topics/admin/tutorials/ansible/tutorial.md

@@ -648,7 +648,7 @@ Sometimes a role will accomplish 95% of what you need to do, but not everything.
 
 # (Optional) Ansible Vault
 
-Now that you have a small role built up, you might start thinking about deploying larger and more complex services and infrastructure. One last common task we want to cover here is the inclusion of secrets.
+Now that you have a small role built up, you might start thinking about deploying larger and more complex services and infrastructure. One last common task we want to cover here is the inclusion of secrets. Ansible Vault is really useful to include encrypted secrets in your playbook repository.
 
 > ### {% icon hands_on %} Hands-on: Setting up secrets
 >
@@ -728,7 +728,7 @@ Now that you have a small role built up, you might start thinking about deployin
 >
 {: .hands_on}
 
-Ansible Vault is really useful to include encrypted secrets in your playbook repository. In real life scenarios where you are sharing your playbooks publicly, be sure to encrypt all secrets from the start (or fix/remove the git history if you ever did.) If you are storing your vault password in a file, remember to add it to your `.gitignore` (or VCS appropriate file.)
+In real life scenarios where you are sharing your playbooks publicly, be sure to encrypt all secrets from the start (or fix/remove the git history if you ever did.) If you are storing your vault password in a file, remember to add it to your `.gitignore` (or VCS appropriate file.)
 
 # Other Stuff
 

topics/admin/tutorials/connect-to-compute-cluster/slides.html

@@ -296,7 +296,7 @@
 
 ---
 
-## Dependency Resolution
+## Tool Dependency Resolution
 
-- Galaxy can resolve dependencies from Conda, Singularity, Docker, and other systems like Modules
-- You can read more about this in the [Toolshed slides](../toolshed/slides.html)
+- Galaxy can resolve dependencies using Conda, Singularity, Docker, and other systems like Modules
+- You can read more about this in the [Galaxy Tool Management with Ephemeris slides](../tool-management/slides.html)

topics/admin/tutorials/tool-management/slides.html

@@ -5,67 +5,74 @@
 - /topics/admin/tutorials/tool-install/slides
 
 
-title: "Ephemeris for Galaxy Tool Management"
+title: "Galaxy Tool Management with Ephemeris"
 questions:
   - How to install, update, and maintain Galaxy tools?
   - How to extract a list of tools from a workflow or Galaxy instance?
+  - How are tool dependencies resolved by Galaxy?
 objectives:
-  - Learn about Ephemeris
   - Extract a list of tools from a workflow
   - Install these tools into a given Galaxy
+  - Understand the tool dependency resolution process
 key_points:
   - Ephemeris and automation help with the tool management on Galaxy
   - There are tool management best practices you can learn from
-  - Do not manage your Galaxy's tools manually
+  - Dependency resolvers have different priorites
 contributors:
   - mvdbeek
   - martenson
   - slugger70
   - hexylena
+  - nsoranzo
 ---
 
-# Galaxy Vocabulary
+## Galaxy Vocabulary
 
 * `tool` or `tool wrapper` - XML file that describes to Galaxy how the underlying software works.
 * `repository` - Versioned code archive in Tool Shed containing Galaxy tool(s).
+* `requirement` or `dependency` - the underlying software needed to execute the tool command
 
 ---
+class: left
 
 # Ways to add tools
 
 You can add tools to Galaxy either
 
-* Manually - Trivial for trivial tools and useful for tool development.
-* Using Tool Shed
-  * Through admin UI in Galaxy
+* Manually - useful for tool development.
+* From the Tool Shed
+  * Through the admin UI in Galaxy
   * Using **ephemeris** (recommended)
 
 ---
 
-# How to add tools manually
+## How to add tools manually
 
 - By default Galaxy loads all tools in `tool_conf.xml.sample` into tool panel.
-- To add local tools by hand:
+- To add a local tool by hand:
   - Make a copy `$ cp tool_conf.xml.sample tool_conf.xml`.
-  - Add your tool entries to the `tool_conf.xml`.
-  - Refresh the site
+  - Add your tool entry to the `tool_conf.xml`.
 - To add local tools with the [galaxyproject.galaxy Ansible role][galaxy-role]:
   - Set `galaxy_local_tools`
   - Run the playbook
+- Tool dependencies need to be installed separately, unless
+  `conda_auto_install: true` is set in `galaxy.yml` (not recommended for
+  production)
+
 
 [galaxy-role]: https://galaxy.ansible.com/galaxyproject/galaxy
 
 ---
 
-# How to install from the Tool Shed
+## How to install from the Tool Shed
 
 * Find the repository you want to install.
 * From Galaxy's admin UI find the repo in the connected TS.
 * Install the repo into Galaxy.
 
 ---
 
-# How to install with Ephemeris
+## How to install with Ephemeris
 
 * Find the repository you want to install.
 * Run `shed-tools install` with the repository name
@@ -76,7 +83,7 @@
 
 Small Python library for Galaxy Management
 
-* Install:
+* Can install:
 	- Tools
 	- Reference data
 	- Workflows
@@ -98,7 +105,7 @@
 (use `workflow-to-tools` to extract list of tools from a workflow)
 
 ---
-## Get Galaxy's tool list
+## Get installed tool list for a Galaxy instance
 
 ```console
 get-tool-list [-h] [-v] [-g GALAXY] [-u USER] [-p PASSWORD] [-a API_KEY]
@@ -233,10 +240,9 @@
 
 # More on Tool Management:
 
-- tool installation
-- configuration
-- suites
-- dependencies
+- Tool installation
+- Configuration
+- Dependencies
 
 ---
 
@@ -250,9 +256,11 @@
 [conda]: https://conda.io/
 
 ---
+class: left
 
 ## Example config entry
- one entry in `shed_tool_conf.xml`
+
+One entry in `shed_tool_conf.xml`:
 
 ```xml
 <section id="filter" name="Filter and Sort" version="">
@@ -281,3 +289,136 @@
     <tool id="testtoolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/0.0.1" />
 </section>
 ```
+
+---
+
+## Tool dependency resolution
+
+* We aim to make Galaxy resolver-independent but Conda-oriented.
+* What resolver is going to be used for the tool dependencies is determined at runtime
+according to the order specified in `dependency_resolvers_conf.xml`.
+
+```xml
+<dependency_resolvers>
+  <tool_shed_packages />
+  <galaxy_packages />
+  <conda />
+  <galaxy_packages versionless="true" />
+  <conda versionless="true" />
+<!-- other resolvers
+  <lmod />
+  <lmod versionless="true" />
+  <modules modulecmd="/opt/Modules/3.2.9/bin/modulecmd" />
+  <modules modulecmd="/opt/Modules/3.2.9/bin/modulecmd" versionless="true" default_indicator="default" />
+  <tool_shed_tap />
+  <homebrew />
+-->
+</dependency_resolvers>
+```
+
+[Full documentation](https://docs.galaxyproject.org/en/master/admin/dependency_resolvers.html)
+
+---
+class: left
+
+## Dependency resolution example
+
+Tool XML contains:
+
+```xml
+<tool id="foo">
+    <requirements>
+        <requirement type="package" version="4.2">bar</requirement>
+    </requirements>
+</tool>
+```
+
+---
+class: left
+
+## `<tool_shed_packages />`
+
+Did tool `foo` depend on a Tool Shed package providing `bar` 4.2 when installed?
+- Yes: Query install DB for `repo_owner`, `repo_name`, `changeset_id`, then source:
+  - `<tool_dependencies_dir>/bar/4.2/<repo_owner>/<repo_name>/<changeset_id>/env.sh`
+- No: continue
+
+---
+class: left
+
+## `<galaxy_packages />`
+
+Does `<tool_dependency_dir>/bar/4.2/env.sh` exist?:
+- Yes: source it
+- No: Does `<tool_dependency_dir>/bar/4.2/bin/` exist?:
+  - Yes: prepend it to `$PATH`
+  - No: continue
+
+Good for dependencies manually installed by an administrator
+
+---
+class: left
+
+## `<conda />`
+
+Does `bar==4.2` exist in Conda?:
+- Yes: use `conda create` to create a Conda env in `<conda_prefix>/envs/` (if it doesn't exist already), then activate it
+- No: continue
+
+[Conda for tool dependencies FAQ](https://docs.galaxyproject.org/en/master/admin/conda_faq.html)
+
+---
+class: left
+
+## `<galaxy_packages versionless="true" />`
+
+Does `<tool_dependency_dir>/bar/default/` exist and is it a symlink?:
+
+- Yes: Does `<tool_dependency_dir>/bar/default/env.sh` exist?:
+  - Yes: source it
+  - No: Does `<tool_dependency_dir>/bar/default/bin/` exist?:
+      - Yes: prepend it to `$PATH`
+      - No: continue
+- No: continue
+
+---
+class: left
+
+## `<conda versionless="true" />`
+
+Does `bar` (any version) exist in Conda?:
+- Yes: use `conda create` to create env in job working directory
+- No: continue
+
+---
+class: left
+
+## `<modules modulecmd="..." />`
+
+Is `bar/4.2` a registered module?:
+- Yes: Load module `bar/4.2`
+- No: continue
+
+See: [Environment Modules](http://modules.sourceforge.net/)
+
+---
+class: left
+
+## `<modules modulecmd="..." versionless="true" />`
+
+Is `bar/default` a registered module?:
+- Yes: Load module `bar/default`
+- No: continue
+
+---
+
+## If dependency is not resolved
+
+Submit the job anyway, maybe it's available on `$PATH`
+
+---
+
+## Using containers for Tool Dependencies
+
+- Galaxy can also use **Docker** or **Singularity** containers to resolve dependencies
+- You can read more about this in the [Tool Dependencies and Containers slides](../../../dev/tutorials/containers/slides.html)

topics/admin/tutorials/tool-management/tutorial.md

@@ -1,7 +1,7 @@
 ---
 layout: tutorial_hands_on
 
-title: "Ephemeris for Galaxy Tool Management"
+title: "Galaxy Tool Management with Ephemeris"
 questions:
   - How to install, update, and maintain Galaxy tools?
   - How to extract a list of tools from a workflow or Galaxy instance?

topics/admin/tutorials/toolshed/slides.html

@@ -171,30 +171,8 @@
 </requirements>
 ```
 
-Galaxy aims to be dependency resolution agnostic.
-
----
-## Dependency resolution
-
-* We aim to make Galaxy resolver-independent but Conda-oriented.
-* What resolver is going to be used for the tool dependency is determined at runtime
-and prioritised in the config file `dependency_resolvers_conf.xml`.
-
-```xml
-<dependency_resolvers>
-  <tool_shed_packages />
-  <galaxy_packages />
-  <conda />
-  <galaxy_packages versionless="true" />
-  <conda versionless="true" />
-  <modules modulecmd="/opt/Modules/3.2.9/bin/modulecmd" />
-  <modules modulecmd="/opt/Modules/3.2.9/bin/modulecmd" versionless="true" default_indicator="default" />
-<!-- other resolvers
-  <tool_shed_tap />
-  <homebrew />
--->
-</dependency_resolvers>
-```
+- Galaxy can resolve dependencies using Conda, Singularity, Docker, and other systems like Modules
+- You can read more about this in the [Galaxy Tool Management with Ephemeris slides](../tool-management/slides.html)
 
 ---
 

topics/dev/tutorials/tool-integration/slides.html

@@ -843,11 +843,13 @@
 
 ## `configfiles`
 
-Like `command`, uses Cheetah code to write:
+A `configfile` creates a text file which can then be used inside the `command` as:
 
-- A config file
-- A whole script or a module
-- Any text file needed for the tool execution
+- A script or a module
+- A file needed to run the tool (e.g. a config file)
+
+Cheetah code and param/output variables can be used inside `configfile` (like
+inside `command`).
 
 ---