Clean up option fail when using AWS Batch (and likely other clouds)

[pditommaso]

Bug report

Nextflow fails to perform the pipeline work directory cleanup when using AWS Batch and other cloud providers

The following error is reported

Feb-14 18:55:04.981 [main] WARN  nextflow.file.FileHelper - Unable to start plugin 'nf-amazon' required by s3://nextflow-ci/work/79/17395af5457010376510e5808a2881
java.lang.NullPointerException: Cannot invoke "nextflow.plugin.CustomPluginManager.getPlugin(String)" because "this.manager" is null
	at nextflow.plugin.PluginsFacade.isStarted(PluginsFacade.groovy:338)
	at nextflow.plugin.PluginsFacade.startIfMissing(PluginsFacade.groovy:432)
	at nextflow.plugin.Plugins.startIfMissing(Plugins.groovy:91)
	at nextflow.file.FileHelper.autoStartMissingPlugin(FileHelper.groovy:288)
	at nextflow.file.FileHelper.asPath(FileHelper.groovy:268)

Complete log file is attached

nextflow.log

[BrunoGrandePhD]

We're running into high S3 storage costs due to our inability to smartly delete files when a workflow completes successfully. It's introducing overhead on Tower admins because we need to manually delete files when buckets get too large.

Could we see this feature bumped in priority? Ideally with a timeline. Thanks!

[bentsherman]

The problem is pretty straightforward, in ScriptRunner:

    protected shutdown() {
        session.destroy()
        session.cleanup()
        Global.cleanUp()
        log.debug "> Execution complete -- Goodbye"
    }

Session destroy() is called before cleanup(). But the Session destroy() shuts down the plugin manager, which breaks the cleanup.

On the other hand, destroy() also shuts down the publish dir executor, which includes waiting for any ongoing publish tasks to finish. So we can't just switch the two calls because then the task directories might be deleted before they are done publishing.

What I propose is to put the cleanup() code in destroy() so that it is called after the publishing but before the plugin manager shutdown. I will draft a PR.

[BrunoGrandePhD]

@bentsherman Thanks for tackling this!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.