Skip to content

Commit

Permalink
Add creating bucket and running workflow section
Browse files Browse the repository at this point in the history
  • Loading branch information
sanjaysrikakulam committed Sep 26, 2024
1 parent 1b52821 commit 95a911c
Show file tree
Hide file tree
Showing 9 changed files with 44 additions and 2 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
46 changes: 44 additions & 2 deletions content/news/2024-09-20-esg-byos-im/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ main_subsite: eu

The [EuroScienceGateway Project](../../projects/esg/) is streamlining the way that users *Bring their Own Storage (BYOS)* to Galaxy. This post covers a specific case in which a user has access to storage resources in the [EGI Federated Cloud](https://www.egi.eu/service/cloud-compute/) and wants to connect it to the [EU Galaxy instance](https://usegalaxy.eu/). However, similar steps can be followed to connect a different Galaxy instance with storage capacity using [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Amazon Web Services S3 Storage](https://aws.amazon.com/s3/), and [Google Cloud Storage](https://cloud.google.com/storage).

In order to connect [Galaxy](https://usegalaxy.eu/) with external cloud storage, the user needs to click on the `User` drop-down menu, select `Preferences` and then `Manage Your Storage Locations`. This will display a list of the cloud storage services already connected to Galaxy. The first time it will be empty and the user can connect to a new cloud storage service clicking on `Create`. For options are given to the user:
In order to connect [Galaxy](https://usegalaxy.eu/) with external cloud storage, the user needs to click on the `User` drop-down menu, select `Preferences` and then `Manage Your Storage Locations`. This will display a list of the cloud storage services already connected to Galaxy. The first time it will be empty and the user can connect to a new cloud storage service clicking on `Create`. The following options are given to the user:

<img class="center" src="byos-create.png" width="80%" alt="Connect Galaxy with cloud storage.">

Expand Down Expand Up @@ -48,7 +48,7 @@ Below are the steps that a user needs to follow to make use of computing resourc

1. Once you are a member of a VO, [configure credentials in Infrastructure Manager](https://docs.egi.eu/users/compute/orchestration/im/dashboard/#cloud-credentials).

1. In [Infrastructure Manager](https://im.egi.eu/) select Select `Deploy a VM` and click `Configure`.
1. In [Infrastructure Manager](https://im.egi.eu/) select `Deploy a VM` and click `Configure`.

1. Select `MinIO` and click `Add`.

Expand Down Expand Up @@ -93,3 +93,45 @@ Along with the URL for the MinIO API endpoint, the user will also need to genera

<img class="center" src="esg-im-minio-07.png" width="80%" alt="MinIO configuration IM.">

# Adding MinIO bucket to Galaxy

Once the MinIO service is up and running the user can create a bucket in MinIO. This is done by accessing the MinIO Console URL and logging in with the access and secret key generated earlier. The user can then create a bucket by clicking on the `Create Bucket` button in the top right corner of the screen and adding a name for the bucket:

<img class="center" src="minio-create-bucket.png" width="80%" alt="Create a bucket in MinIO.">

The user can then add this bucket to Galaxy by following the steps described at the beginning of this post (Select `Any S3 Compatible Storage` option from the available templates). The user needs to provide a `Name`, `Access Key ID`, `Bucket` (name of the bucket), the `S3-Compatible API Endpoint` (MinIO API endpoint), the `Secret Access Key` and submit the form.

<img class="center" src="byos-add-minio-bucket-to-galaxy.png" width="80%" alt="Add your MinIO bucket to Galaxy.">

Once the bucket is added to Galaxy the user can start using it as the default storage location or select it at different levels: per `History`, per `Tool`, and `Workflow`.

<img class="center" src="byos-your-newly-added-external-storage.png" width="80%" alt="Newly added external storage.">

# Running workflows in Galaxy with Bring Your Own Storage

The user can now run tools and workflows in Galaxy using the newly configured external storage. Below is an example of a workflow that performs the assembly of bacterial genomes using [Shovill](https://usegalaxy.eu/?tool_id=toolshed.g2.bx.psu.edu%2Frepos%2Fiuc%2Fshovill%2Fshovill%2F1.1.0%20galaxy2) a faster _de novo_ assembly pipeline for Illumina paired end reads.

<iframe title="Galaxy Workflow Embed" style="width: 100%; height: 700px; border: none;" src="https://usegalaxy.eu/published/workflow?id=b17603457e1d4262&embed=true&buttons=true&about=true&heading=true&minimap=true&zoom_controls=true&initialX=-20&initialY=-20&zoom=1"></iframe>

## Steps to run the workflow:
1. Create a new history in Galaxy.
2. Select the newly added external storage as the preferred storage location for the `History`.

<img class="center" src="byos-history-level-selection.png" width="80%" alt="Set storage for this history.">

3. Upload / Import the [input data](https://usegalaxy.eu/published/history?id=8ac959df218c9188).
4. Go to the `Workflow` tab and select the workflow and run it.

<img class="center" src="run-galaxy-workflow.png" width="80%" alt="Set storage for this history.">

5. Once the workflow is completed,

<img class="center" src="successful-completion-of-the-workflow-invocation.png" width="80%" alt="Successful completion of workflow run">

the user can find all the output files in the external storage.

<img class="center" src="all-workflow-datasets-now-in-the-bucket.png" width="80%" alt="Output files in external storage.">

6. The user can also evaluate it by clicking the `Dataset details` of a job and observe that the user's external storage is used as the `Dataset Storage`

<img class="center" src="dataset-storage-location.png" width="80%" alt="Dataset details showing external storage.">
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 95a911c

Please sign in to comment.