Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename files before upload to AWS nodes #37

Open
pbiology opened this issue Nov 18, 2021 · 0 comments
Open

Rename files before upload to AWS nodes #37

pbiology opened this issue Nov 18, 2021 · 0 comments
Labels
enhancement New feature or request

Comments

@pbiology
Copy link

What needs to be done:
In order to avoid issues with sensitive information being sent to cloud, we could do a pre-step with a local executor. If we re-name files locally, keep a dictionary in memory and then re-name the files again once they return from the cloud nodes, then we can avoid sending labIDs to the cloud.

Suggestions on how to get it done:
Before the rest of the workflow is executed, run a step with a local executor for file renaming. One perhaps need to be a little careful not to use up all resources on the login node.

What are the arguments for getting it done:
We can side-step some of the legal issues hopefully

Task is considered finished when:

@pbiology pbiology added the enhancement New feature or request label Nov 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant