Update to 0.6.3 fails

[stav-ros]

Hey, I just tried to update to the latest version and I get the following:

2024-03-07 16:22:04 URL:https://nextpyp.app/files/pyp/0.6.3/workflows/class_tutorial.toml [7248/7248] -> "/opt/nextPYP/workflows/class_tutorial.toml" [1]
Created local directory: /opt/nextPYP/local
Created shared directory: /opt/nextPYP/shared
Using existing configuration file: /opt/nextPYP/config.toml
Installing integrations with systemd ...
Job for nextPYP.service failed because the control process exited with error code.
See "systemctl status nextPYP.service" and "journalctl -xeu nextPYP.service" for details.

When checking "systemctl status nextPYP.service" I see the follwing:

mar 07 16:27:30 cryo1 systemd[1]: Stopped nextPYP.service - nextPYP.
mar 07 16:27:30 cryo1 systemd[1]: nextPYP.service: Consumed 1.061s CPU time.
mar 07 16:27:30 cryo1 systemd[1]: nextPYP.service: Start request repeated too quickly.
mar 07 16:27:30 cryo1 systemd[1]: nextPYP.service: Failed with result 'exit-code'.
mar 07 16:27:30 cryo1 systemd[1]: Failed to start nextPYP.service - nextPYP.

Logs for micromon or others don't show any relevant information. Where could I look to find the issue?

[stav-ros]

Update: There was an issue in config.toml where I added a [standalone] section to define the number of GPUs as per the config https://nextpyp.app/files/pyp/latest/docs/reference/config.html

The reason is that it does not detect my 4090 GPU for some reason when I run motioncorr3

```

Progress: 41it [00:01, 25.44it/s]
60 2024-03-07 16:54:45 [INFO] /opt/pyp/bin/run/pyp:57 | Total time in tomo_swarm 00h 00m 04s seconds
61 2024-03-07 16:54:45 [INFO] /opt/pyp/bin/run/pyp:3178 | Filesystem Size Used Avail Use% Mounted on
62 2024-03-07 16:54:45 [INFO] /opt/pyp/bin/run/pyp:3178 | /dev/sda1 3.6T 314G 3.1T 9% /media/data/
63 /usr/local/envs/pyp/lib/python3.8/site-packages/torch/cuda/init.py:107: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:109.)
64 return torch._C.cuda_getDeviceCount() > 0
65 joblib.externals.loky.process_executor.RemoteTraceback:
66 """
67 Traceback (most recent call last):
68 File "/usr/local/envs/pyp/lib/python3.8/shutil.py", line 791, in move
69 os.rename(src, real_dst)
70 FileNotFoundError: [Errno 2] No such file or directory: 'TS_01_039-60.0.mrc' -> '../TS_01_039-60.0.avg'
71
72 During handling of the above exception, another exception occurred:
73
74 Traceback (most recent call last):
75 File "/usr/local/envs/pyp/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 428, in _process_worker
76 r = call_item()
77 File "/usr/local/envs/pyp/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 275, in call
78 return self.fn(*self.args, **self.kwargs)
79 File "/usr/local/envs/pyp/lib/python3.8/site-packages/joblib/parallel_backends.py", line 620, in call
80 return self.func(*args, **kwargs)
81 File "/usr/local/envs/pyp/lib/python3.8/site-packages/joblib/parallel.py", line 288, in call
82 return [func(*args, **kwargs)
83 File "/usr/local/envs/pyp/lib/python3.8/site-packages/joblib/parallel.py", line 288, in
84 return [func(*args, **kwargs)
85 File "/opt/pyp/src/pyp/system/mpi.py", line 174, in wrapper
86 func(*arg)
87 File "/opt/pyp/src/pyp/align/core.py", line 4583, in align_movie_super
88 shutil.move( name + ".mrc", f"../{aligned_average}")
89 File "/usr/local/envs/pyp/lib/python3.8/shutil.py", line 811, in move
90 copy_function(src, real_dst)
91 File "/usr/local/envs/pyp/lib/python3.8/shutil.py", line 435, in copy2
92 copyfile(src, dst, follow_symlinks=follow_symlinks)
93 File "/usr/local/envs/pyp/lib/python3.8/shutil.py", line 264, in copyfile
94 with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
95 FileNotFoundError: [Errno 2] No such file or directory: 'TS_01_039-60.0.mrc'
96 """
97
98 The above exception was the direct cause of the following exception:
99
100 Traceback (most recent call last):
101 File "/opt/pyp/bin/run/pyp", line 3448, in
102 tomo_swarm(args.path, args.file, args.debug, args.keep, args.skip)
103 File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
104 return func(*args, **kwargs)
105 File "/opt/pyp/bin/run/pyp", line 1446, in tomo_swarm
106 ] = preprocess.read_tilt_series(
107 File "/opt/pyp/src/pyp/preprocess/core.py", line 676, in read_tilt_series
108 mpi.submit_function_to_workers(
109 File "/opt/pyp/src/pyp/system/mpi.py", line 185, in submit_function_to_workers
110 parallel(delayed(wrapper)(func, *arg, current_directory=current_directory) for idx, func in enumerate(funcs) for arg in args[idx])
111 File "/usr/local/envs/pyp/lib/python3.8/site-packages/joblib/parallel.py", line 1098, in call
112 self.retrieve()
113 File "/usr/local/envs/pyp/lib/python3.8/site-packages/joblib/parallel.py", line 975, in retrieve
114 self._output.extend(job.get(timeout=self.timeout))
115 File "/usr/local/envs/pyp/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
116 return future.result(timeout=timeout)
117 File "/usr/local/envs/pyp/lib/python3.8/concurrent/futures/_base.py", line 444, in result
118 return self.__get_result()
119 File "/usr/local/envs/pyp/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
120 raise self.exception
121 FileNotFoundError: [Errno 2] No such file or directory: 'TS_01_039-60.0.mrc'
122 2024-03-07 16:54:45 [ERROR] /opt/pyp/bin/run/pyp:3453 | PYP (tomoswarm) failed
123

[abartesaghi]

Does the split job in the Jobs panel show Split (gpu) when running?

[stav-ros]

Yes. Is there a way to test whether cuda/gpu was detected correctly?

[abartesaghi]

If you see the (gpu) label in the Split job name, that means the GPU resource was requested correctly by nextPYP.

A simple way test if the GPUs are visible is to run:

# cd to the nextpyp installation folder
apptainer exec --nv pyp.sif nvidia-smi

[stav-ros]

If you see the (gpu) label in the Split job name, that means the GPU resource was requested correctly by nextPYP.

A simple way test if the GPUs are visible is to run:

# cd to the nextpyp installation folder
apptainer exec --nv pyp.sif nvidia-smi

Thanks, that shows nvidia-smi running cuda 12.2 so I guess that it is at least detected.

[abartesaghi]

@stav-ros, we may have found the reason for this problem and are currently testing a solution. We will release the changes as part of the next patch version.

[abartesaghi]

This was addressed in version 0.6.4.