python3: can't open file 'dnaPipeTE.py': [Errno 2] No such file or directory

[AntonioBaeza]

Hi Clement. The owner of the server/cluster did update singularity. I am using version v. 3.5.3. Not sure if installed well and the error message is related to dnaPipeTE.py this time. I checked for information about how to resolve this problem but nothing. Thank you.

(base) ant@hillary:$ pwd
/home/ant
(base) ant@hillary:$
(base) ant@hillary:$ mkdir /dnaPipeTE
(base) ant@hillary:$ ls
dnaPipeTE
(base) ant@hillary:$
(base) ant@hillary:$ cd /dnaPipeTE
(base) ant@hillary:/dnaPipeTE$ pwd
/home/ant/dnaPipeTE
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$ singularity version
3.5.3
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$ singularity pull --name dnapipete.img docker://clemgoub/dnapipete:latest
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob bff3e048017e done
Copying blob 7454a22fef40 done
Copying blob 48fbb22cd9b4 done
Copying blob e2d9047f3ead done
Copying blob 8a8adba02010 done
Copying blob ad67bcb30b6a done
Copying blob 8ac634985821 done
Copying blob 70d11079cde6 done
Copying blob fbff193b270a done
Copying blob bb765d177084 done
Copying blob 68e8af20fafd done
Copying blob 281a7872f45e done
Copying blob 3adbf49e1bc4 done
Copying blob b285af44ce59 done
Copying blob bc37b2c348d0 done
Copying blob d3fd4fc0d24a done
Copying blob ce2e9287fada done
Copying blob 57002df47231 done
Copying blob 54b6d5886e3b done
Copying blob 48a0644ebad1 done
Copying blob c564aa69f587 done
Copying blob bf9e261ceb8d done
Copying blob b776c6dddc75 done
Copying blob 045f6fa5b83b done
Copying blob 9e53910bf641 done
Copying blob 13d2c9d56b9d done
Copying blob 5042d3630199 done
Copying blob 443adbe8f68a done
Copying blob 9942e1d21344 done
Copying blob 4f4fb700ef54 done
Copying config c7f39cf116 done
Writing manifest to image destination
Storing signatures
2023/02/21 19:37:49 info unpack layer: sha256:bff3e048017eab8adaed886bad4b3c5c81f7d952b65b056dca8d6dbc198222dd
2023/02/21 19:37:49 warn rootless{usr/local/man} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:37:50 info unpack layer: sha256:7454a22fef406b7c64195cfecc1f98bbf05d806c2fe8b2c090611d8174344152
2023/02/21 19:38:02 info unpack layer: sha256:48fbb22cd9b4218cb147f8c9289504463fab6fc409dd978ed2a41c74295209f1
2023/02/21 19:38:05 info unpack layer: sha256:e2d9047f3ead0e6275c786ee0c88173e9fda0a3c429090abd1a801459cf6318d
2023/02/21 19:38:16 info unpack layer: sha256:8a8adba02010309fedc1063050c516e399a57c40d0e82e6c6c353f345024a4de
2023/02/21 19:38:21 info unpack layer: sha256:ad67bcb30b6af5dabfa475f73bbe1aaf1598cd24db9260aec001a16fbecbdfa2
2023/02/21 19:38:21 warn rootless{opt/src/hmmer-3.3.2/easel/documentation/figures/easel_techtree.pdf} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:38:22 info unpack layer: sha256:8ac634985821af7eea08a346f75053095514bad54749e5e2c2c1bb2beaddacad
2023/02/21 19:38:22 info unpack layer: sha256:70d11079cde67fb348435bfea1792e18b378183d7b2eb977551c2b0ac72c73ad
2023/02/21 19:39:03 info unpack layer: sha256:fbff193b270a70ae7eba474c4cdb74c717a225b28809fff0752d6e9ee262257e
2023/02/21 19:39:03 info unpack layer: sha256:bb765d177084f78bcc6ef19efec1e93505f7c503fab47ed8bd895f492d8bec30
2023/02/21 19:39:06 info unpack layer: sha256:68e8af20fafdc03b0c56c2a6b889be17bce6d1b78907f11416e07e42cabf3430
2023/02/21 19:39:08 info unpack layer: sha256:281a7872f45e4f1913a29ecdd7a25aac17bf0ff2cef68d2b65791b4d94c87913
2023/02/21 19:39:12 info unpack layer: sha256:3adbf49e1bc415c1cf93ce9d8bba2088795be2d5ff33a0f7664b7f36e2c24130
2023/02/21 19:39:12 info unpack layer: sha256:b285af44ce591fab52be4c2fc1586000d04484ed6d14f8cf8652706d146067e4
2023/02/21 19:39:13 info unpack layer: sha256:bc37b2c348d0bbcd24aba45a4419f1a9b9cc21a3ee28df50f719cd4ebc8b6c22
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/KernSmooth.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/MASS.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/Matrix.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/boot.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/class.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/cluster.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/codetools.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/foreign.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/lattice.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/mgcv.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/nlme.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/nnet.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/rpart.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/spatial.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:15 warn rootless{R-4.2.1/src/library/Recommended/survival.tgz} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:17 warn rootless{R-4.2.1/tests/Pkgs/pkgA} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2023/02/21 19:39:19 info unpack layer: sha256:d3fd4fc0d24a0ab3ece78ae9609f71e5830fc6a2ea602f69c6f2405705175bff
2023/02/21 19:39:23 info unpack layer: sha256:ce2e9287fadac9a414fd1a74ed7e9e17dc7c7ded15e66bc9aaa7c4c1ba1f7c8e
2023/02/21 19:39:24 info unpack layer: sha256:57002df472312a576204b28a72ca44d6112cd9dfebef20f24a59ab7d6c0cf066
2023/02/21 19:39:24 info unpack layer: sha256:54b6d5886e3b906f0e69430a6e336d600fb226a38b530d0983877a56c380f5d3
2023/02/21 19:39:24 info unpack layer: sha256:48a0644ebad1664152f33805f5627d4cbf0f9919dec5347c467a95ae5b5318ee
2023/02/21 19:39:24 info unpack layer: sha256:c564aa69f587d709463c3bbe4105df3e933949db563345a3689e2c841c4b74cc
2023/02/21 19:39:24 info unpack layer: sha256:bf9e261ceb8d24afcf0b1733e7ec371cc5f1f97912113daebc535a5505eee9e2
2023/02/21 19:39:24 info unpack layer: sha256:b776c6dddc75f0a6191b4c30af093a50a9c6b40234a94ffc237a27f3071c18f0
2023/02/21 19:39:24 info unpack layer: sha256:045f6fa5b83ba7849563963a0a93136d2348ba887b89b6c814f5996bb79b23fa
2023/02/21 19:39:24 info unpack layer: sha256:9e53910bf641d9f7ba4e284afb4985084a767bf76e2ff0f269b339b7525462f1
2023/02/21 19:39:24 info unpack layer: sha256:13d2c9d56b9db4b5e0e390e30ab72c3a57b7d65cf070234c569a43ff832b5f9f
2023/02/21 19:39:24 info unpack layer: sha256:5042d36301999af1177e1a5857f318f52badaf23d9316fcc3aff0aadec07913e
2023/02/21 19:39:24 info unpack layer: sha256:443adbe8f68a9093d0e8277eb1b56d24d2f80aa20f0e97253ad945f7bef4039a
2023/02/21 19:39:24 info unpack layer: sha256:9942e1d2134458c4d649b74cbd0da41c14fe9b3469e9439b77b03c2593c9334a
2023/02/21 19:39:24 info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
INFO: Creating SIF file...
INFO: Build complete: dnapipete.img
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$ singularity shell /dnaPipeTE/dnapipete.img
Singularity> python3 dnaPipeTE.py -h
python3: can't open file 'dnaPipeTE.py': [Errno 2] No such file or directory
Singularity> exit
exit
(base) ant@hillary:/dnaPipeTE$ ls
dnapipete.img
(base) ant@hillary:/dnaPipeTE$

[clemgoub]

Hi Antonio,

Can you try with this command instead?:

python3 /opt/dnaPipeTE/dnaPipeTE.py -h

I thought I had changed the default location for the script,... Will check on my side now.

---

To make sure, you are not encountering the error in issue #77 anymore right?

[clemgoub]

The problem just has been reported to me from another user. I think the problem is specific to singularity. Using docker, the default ("landing") directory when opening the container is /opt/dnaPipeTE -- I think this is ignored by singularity, thus why you couldn't find the executable.

It also occurs to me that you also need to cd /opt/dnaPipeTE before executing dnaPipeTE.py.

Let me know if the issue persists!

Cheers,

Clément

[AntonioBaeza]

Thank you. Keep trying. I will keep you posted. Thanks!

[AntonioBaeza]

Trying to find the right steps and syntax, still not working

(base) ant@hillary:$ ls
Bats2021 famdb.py LemonSharkGSS Madracis2023 Metapenaeus Panda Project Romerolagus Whales
Cardisoma Gecarcinus LemonSharkRNAseq Mammals miniconda3 PanulirusMX2021 Project2 TiburonToro
Darter grbgetkey LeopardShark master.zip Miniconda3-latest-Linux-x86_64.sh Pejelagarto Remipedia TigerAmur
dnaPipeTE grbprobe licensetools9.5.1_linux64.tar.gz Meduza MitoPhAST-master PlantNanopore RESPECT Tridacna
EDTA grb_ts LysmataRNA Merluza MitoPhAST-master.zip Plongipes R_libs Turtle
FAMDB Hyppolite Madracis MetaGenomeMito Oikopleura Ppascuensis RNAmmer VampBat
(base) ant@hillary:$
(base) ant@hillary:$
(base) ant@hillary:$ cd dnaPipeTE/
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$ pwd
/home/ant/dnaPipeTE
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$ singularity shell --bind Project2:/mnt /dnaPipeTE/dnapipete.img
FATAL: container creation failed: mount /home/ant/dnaPipeTE/Project2->/mnt error: while mounting /home/ant/dnaPipeTE/Project2: mount source /home/ant/dnaPipeTE/Project2 doesn't exist
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$ singularity shell /dnaPipeTE/dnapipete.img
Singularity>
Singularity>
Singularity> singularity shell --bind Project2:/mnt /dnaPipeTE/dnapipete.img
bash: singularity: command not found
Singularity>
Singularity> exit
exit
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$ singularity shell --bind Project2:/mnt /opt/dnaPipeTE/dnapipete.img
FATAL: could not open image /opt/dnaPipeTE/dnapipete.img: failed to retrieve path for /opt/dnaPipeTE/dnapipete.img: lstat /opt/dnaPipeTE: no such file or directory
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$ singularity shell --bind home/ant/Project2:/mnt /opt/dnaPipeTE/dnapipete.img
FATAL: could not open image /opt/dnaPipeTE/dnapipete.img: failed to retrieve path for /opt/dnaPipeTE/dnapipete.img: lstat /opt/dnaPipeTE: no such file or directory
(base) ant@hillary:/dnaPipeTE$ ^C
(base) ant@hillary:/dnaPipeTE$ ls
config.ini Dfam.h5 Dfam.h5.gz dnapipete.img famdb.py library.fasta
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$ cd /opt/dnaPipeTE
-bash: cd: /opt/dnaPipeTE: No such file or directory
(base) ant@hillary:/dnaPipeTE$
(base) ant@hillary:/dnaPipeTE$ singularity shell ~/dnaPipeTE/dnapipete.img
Singularity>
Singularity> cd /opt/dnaPipeTE
Singularity>
Singularity> singularity shell --bind ~Project2:/mnt ~/dnaPipeTE/dnapipete.img
bash: singularity: command not found
Singularity>
Singularity> ls
README.md blast_sorter.py dnaPipeTE.py fixjava.sh init.sh new_list_of_RM_superclass_colors_sortedOK pieColors test_config.sh
bin config.ini dnaPipefront.png graph.R landscapes.R pieChart.R test
Singularity>
Singularity>
Singularity> singularity shell --bind ~Project2:/mnt /opt/dnaPipeTE/dnapipete.img
bash: singularity: command not found
Singularity>
Singularity>
Singularity> python3 dnaPipeTE.py -input /mnt/data/uncseqs_2.fastq -output /mnt/dnaPipeTE_proj2 -genome_size 3000000000 -genome_coverage 0.15 -sample_number 2 -RM_lib nt/data/library14.fasta -RM_t 0.25 -cpu 10
_ _____ _ _______ ______
| | | __ () |__ | ____|
| | __ __ _| |) | _ __ | | | |
/ | '_ \ / _ | / | ' \ / _ \ | | __|
| (| | | | | (| | | | | |) | / | | |
_,|| ||_,|| || ./ _|| |_____|
| |
||

 De Novo Anssembly and Annotation PIPEline for Transposable Elements
                  v.1.4c "container" 06.Oct.2022

                                                                                                                                                         

                                             :@@@@'
                                           ,@@@@@@@@;
                                          +@@@@@@@@@@,
                                        .@@@@@@@@@@@@#
                                +@#.   '@@@@@@@@@@@@@#
                               #@@@@+`@@@@@@@@@@@@@@@,
                               '@@@@@@@@@@@@@@@@@@@@'
                                :@@@@@@@@@@@@@@@@@@.
                                 :@@@@@@@@@@@@@@@+
                                #@@@@@@@@@;@@@@@,
                              ,@@@#`.@@@@@@#,@#
                             +@@@:    +@@@@@@,
                           .@@@@       #@@@@@@#
                          '@@@;       '@@@@@@@@@
                        `@@@@`      `@@@@`:@@@@@
                       ;@@@'       ;@@@'    +@+
                      @@@@.       @@@@.
                    :@@@#       :@@@#
                   #@@@,       #@@@,
                 ,@@@#`@@'   ,@@@#
                +@@@:'@@@@@:+@@@:
              .@@@@`@@@@@@@@@@@
             '@@@;:@@@@@@@@@@;
            #@@@`#@@@@@@@@@@`
           ,@@@;@@@@@@@@@@+
           #@@@@@@@@@@@@@.
           @@@@@@@@@@@@#
          .@@@@@@@@@@@,
         +@@@@@@@@@@#
         @@@@@@#+;.
          +@@;

       Let's go !!!

Start time: Sun Mar 12 18:28:38 2023
Traceback (most recent call last):
File "dnaPipeTE.py", line 695, in
Sampler = FastqSamplerToFasta(args.input_file, args.sample_size, args.genome_size, args.genome_coverage, args.sample_number, args.output_folder, False)
File "dnaPipeTE.py", line 132, in init
os.makedirs(self.output_folder)
File "/usr/lib/python3.5/os.py", line 241, in makedirs
mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/mnt/dnaPipeTE_proj2'
Singularity>
Singularity>
Singularity> shell --bind ~Project2:/mnt /opt/dnaPipeTE/dnapipete.img
bash: shell: command not found
Singularity>
Singularity> singularity shell --bind ~Project2:/mnt /opt/dnaPipeTE/dnapipete.img
bash: singularity: command not found
Singularity>
Singularity>

[clemgoub]

Hello Antonio,

You were actually very close I believe. I'm sorry about the troubles. Using container isn't always straightforward. I hope these details will help!

The singularity image works more or less like a virtual machine within your local (physical) machine. Once in the image, you can't access the local computer unless you mount local directories to it. This is what is done with --bind.

There are two ways of running a Singularity image: either by submitting a command to it and waiting for the output (singularity exec) or interactively, by "going in" the image (singularity shell). I find the second way simpler to begin with.

Singularity commands are executed on your local machine. Once in the image runs, the command are only interpreted by the virtual machine (the image). Using the interactive mode (singularity shell), you see that the prompt changes from (base) ant@hillary:~$ to Singularity>. This indicates you are in. At this point, no need to call again Singularity (it is actually not installed in the image). To reach an outside directory (a directory on your local machine) you need to point to it following the bind path you have set with --bind.

Now I'm gonna sample some command you executed above, and in the right order, it should work! Basically, what failed is that when you finally launched the image, you did not mount the local directory, but try to do it later inside the image (that's why it said bash: singularity: command not found).

1. In the local machine: go to your project directory:

cd  dnaPipeTE/

if you want a project folder called Project2, you need to create it before starting the container. We also create a data subfolder that will host the reads.

mkdir Project2
mkdir Project2/data

We now need to copy (or move) the input reads and TE library in this folder ~/dnaPipeTE/Project2/data/

cp <actual_path_to_reads>/uncseqs_2.fastq ~/Project2/data/
cp <actual_path_to_te_library>/library14.fasta ~/Project2/data/

replace <actual_path_to_XXX> with their actual path

Now, let's quickly look at the directory structure. It should be like so

~/dnaPipeTE <== we are currently here
└── dnapipete.img
└── Project2/
           └── data/
                  └── uncseqs_2.fastq
                  └── library14.fasta

we can double check we are at the right place by looking at the prompt. It should be (base) ant@hillary:~/dnaPipeTE$

2. We can now start the image and bind the Project2/ directory to it

singularity shell --bind ~/dnaPipeTE/Project2:/mnt ~/dnaPipeTE/dnapipete.img

In the image, first check that we are in, by looking at the prompt. It should be Singularity>.
Then, the first thing to do is move into the folder that contains the scripts. It is located in /opt/dnaPipeTE (/opt is a directory in the image)

cd /opt/dnaPipeTE
ls

ls should return:

README.md  blast_sorter.py  dnaPipeTE.py      fixjava.sh  init.sh       new_list_of_RM_superclass_colors_sortedOK  pieColors  test_config.sh
bin        config.ini       dnaPipefront.png  graph.R     landscapes.R  pieChart.R                                 test

3. Let's now check that the outside directory (~/Project2) has been mounted properly.

ls /mnt

it should return

data/

You should also be able to locate the reads and TE library:

ls /mnt/data

should return

uncseqs_2.fastq
library14.fasta

Normally, you can't write anything in a folder of the image, except in /mnt, that redirects to ~/Project2.
Now that we have loaded the image, moved to the correct folder (/opt/dnaPipeTE) and mounted the project directory, we are ready to go:

python3 dnaPipeTE.py -input /mnt/data/uncseqs_2.fastq -output /mnt/dnaPipeTE_proj2 -genome_size 3000000000 -genome_coverage 0.15 -sample_number 2 -RM_lib /mnt/data/library14.fasta -RM_t 0.25 -cpu 10

Here, you indicate the output folder to be -output /mnt/dnaPipeTE_proj2. In you local machine, it will be located in ~/dnaPipeTE/Project2/dnaPipeTE_proj2

Once the analysis ends, you can leave the image:

exit

You will see the prompt changing from >Singularity to (base) ant@hillary:~/dnaPipeTE$. You can now navigate your results in ~/dnaPipeTE/Project2/dnaPipeTE_proj2

If you still encounter issues, I'll be happy to do a visio with you so I can directly guide you, it might be the simplest!

Cheers,

Clément

[AntonioBaeza]

Sincerely appreciate Clement. The pipeline is working!