Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

try free runner and increate resource for Github CI test #42

Merged
merged 6 commits into from
May 7, 2024
Merged

try free runner and increate resource for Github CI test #42

merged 6 commits into from
May 7, 2024

Conversation

gq1
Copy link
Member

@gq1 gq1 commented May 7, 2024

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@github-actions GitHub Actions
Copy link

github-actions bot commented May 7, 2024 •
edited
Loading

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 0458cd4

+| ✅ 130 tests passed       |+
#| ❔  19 tests were ignored |#
!| ❗   3 tests had warnings |!

❗ Test warnings:

  • files_exist - File not found: conf/igenomes.config
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in base.config: Customise requirements for specific processes.

❔ Tests ignored:

  • files_exist - File is ignored: assets/nf-core-genomeassembly_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-genomeassembly_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-genomeassembly_logo_dark.png
  • files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
  • files_exist - File is ignored: .github/workflows/awstest.yml
  • files_exist - File is ignored: .github/workflows/awsfulltest.yml
  • nextflow_config - Config variable ignored: manifest.name
  • nextflow_config - Config variable ignored: manifest.homePage
  • files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
  • files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
  • files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
  • files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
  • files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
  • files_unchanged - File does not exist: assets/nf-core-genomeassembly_logo_light.png
  • files_unchanged - File does not exist: docs/images/nf-core-genomeassembly_logo_light.png
  • files_unchanged - File does not exist: docs/images/nf-core-genomeassembly_logo_dark.png
  • files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/genomeassembly/genomeassembly/.github/workflows/awstest.yml

✅ Tests passed:

Run details

  • nf-core/tools version 2.8
  • Run at 2024-05-07 14:33:23

@gq1
Copy link
Member Author

gq1 commented May 7, 2024

Still more storage needed:

    INFO:    Creating SIF file...
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: while creating SIF: while creating container: write /home/runner/.apptainer/cache/oci-tmp/tmp_936536538: copy_file_range: no space left on device

@gq1
Copy link
Member Author

gq1 commented May 7, 2024

After increasing the memory to 15G, there is a bug in this module:

Process `SANGERTOL_GENOMEASSEMBLY:GENOMEASSEMBLY:HIC_MAPPING:BAMTOBED_SORT (baUndUnlc1)` terminated with an error exit status (2)

Command executed:

  samtools view -@4 -u -F0x400 baUndUnlc1_mkdup.bam | bamToBed | sort -k4 --parallel=4 -S 7.5G -T . > baUndUnlc1_merged_sorted.bed

genomeassembly/modules/local/bamtobed_sort.nf

Line 25 in 271885e

def buffer_mem = task.memory.toGiga() / 2

@ksenia-krasheninnikova Can you round the half memory? It seems the sort command only take the whole number as buffer memorey.

@gq1
Copy link
Member Author

gq1 commented May 7, 2024

@ksenia-krasheninnikova I finished my test now.

  1. Your CI testing still needs lots of storage for singularity images, the free version of runner not enough.
  2. The large runner ubuntu2204-4c has 4 CPUs and 16G memory, the same like the free version. You probably can give all of them to nextflow and I hope it will run fast. But my last run took 15mins, comparing your previous run only 12 mins. But still it is good to merge this change.
  3. I think you can fix this when you have time. try free runner and increate resource for Github CI test #42 (comment)

@ksenia-krasheninnikova
Copy link
Contributor

ksenia-krasheninnikova commented May 7, 2024 •
edited
Loading

@gq1: The update looks fine to me. I have added rounding of the buffer memory in bamtobed_sort module.

@gq1
Copy link
Member Author

gq1 commented May 7, 2024

This error can be seen randomly. Re-running should work:

Failed to pull singularity image
  command: singularity pull  --name ghcr.io-nbisweden-fastk_genescopefk_merquryfk-1.2.img.pulling.1715092918304 docker://ghcr.io/nbisweden/fastk_genescopefk_merquryfk:1.2 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:b56b35773cdcbd2913e4d4b4c8deeb38a4f3ff4abf1e22c1f66d7e561739bc3f
    Copying blob sha256:ac77672a979829fb237e3b2bf02d933da8f1419de36efc0e902fdb2c60bf521c
    Copying blob sha256:83eeaff6d4af9433f22fc60ba62bc851474157f4433388e17b76f05f7badb9a8
    Copying blob sha256:8cc533bc8a27c6fa8d5d4ca6804f553f8be9fdd74bd99e5ecfebef9416078cf6
    Copying blob sha256:7b303595d9b321a9020d0ddbf1dea4c83237e2367117606a8d5466c446714ba1
    Copying blob sha256:38a8fd9de21eaee23a103dc924054d05dc4c4bc06334e7930f06871b17fca606
    Copying blob sha256:8faa8459fed85f8afdba73a54844c4e5fd34779577afd8be3d9a38a2e[70](https://github.com/sanger-tol/genomeassembly/actions/runs/8987250188/job/24685687775?pr=42#step:9:71)2e849
    Copying blob sha256:84463ad43d8de19d0f07ae43c081868ab61f920766b8e593ecf87d4cfb6a34fc
    Copying blob sha256:09db92fad269c5e2ffb9526eb8613884f68ea277b173f65cd029bd5fc3d36f6d
    Copying blob sha256:4f4fb700ef54461cfa025[71](https://github.com/sanger-tol/genomeassembly/actions/runs/8987250188/job/24685687775?pr=42#step:9:72)ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
    Copying blob sha256:307f67c1539cc9a801e3126a462c953404bee7e831909a332cd0c2de897[74](https://github.com/sanger-tol/genomeassembly/actions/runs/8987250188/job/24685687775?pr=42#step:9:75)928
    Copying config sha256:af05a1a95bc1bb513db1c2d166970e2ff5[75](https://github.com/sanger-tol/genomeassembly/actions/runs/8987250188/job/24685687775?pr=42#step:9:76)d6de8e80635987a09b74a617f1dd
    Writing manifest to image destination
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: while getting config: no descriptor found for reference "60da47b6[76](https://github.com/sanger-tol/genomeassembly/actions/runs/8987250188/job/24685687775?pr=42#step:9:77)ee2b52be5b774a9b65e54edc2bb2fd319876efd09efd8d99d39ada"

@gq1 gq1 changed the title try free runner try free runner and increate resource for Github CI test May 7, 2024
@gq1 gq1 merged commit f4701f1 into dev May 7, 2024
6 checks passed
@gq1
Copy link
Member Author

gq1 commented May 7, 2024

Another error message:

Command output:
  Looking for mitochondrion for Caradrina clavipalpis

Command error:
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  INFO:    Environment variable SINGULARITYENV_NCBI_API_KEY is set, but APPTAINERENV_NCBI_API_KEY is preferred
  Looking for mitochondrion for Caradrina clavipalpis
  Traceback (most recent call last):
    File "/opt/MitoHiFi/src/findMitoReference.py", line 96, in <module>
      for g in [args.species] + list(get_lineage(args.species)):
    File "/opt/MitoHiFi/src/findMitoReference.py", line 29, in get_lineage
      record = Entrez.read(handle)
    File "/usr/local/lib/python3.6/dist-packages/Bio/Entrez/__init__.py", line 508, in read
      record = handler.read(handle)
    File "/usr/local/lib/python3.6/dist-packages/Bio/Entrez/Parser.py", line 335, in read
      self.parser.ParseFile(handle)
    File "../Modules/pyexpat.c", line 473, in EndElement
    File "/usr/local/lib/python3.6/dist-packages/Bio/Entrez/Parser.py", line 704, in endErrorElementHandler
      raise RuntimeError(value)
  RuntimeError: Search Backend failed:

@muffato muffato deleted the runner-testing branch May 29, 2024 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ksenia-krasheninnikova ksenia-krasheninnikova ksenia-krasheninnikova approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@gq1 @ksenia-krasheninnikova