The particle refinement step is stuck

[Angeevillor]

Hello, I am running the sample data (EMPIAR-10164) according to the tutorial. The previous steps worked fine, but during the particle refinement step, it remained stuck here for a day. I'm not sure what the problem is and how to deal with it. Could you please give me some advice? Thanks!

PS: I am running on a single machine, here are the Settings in my "Resources":

[abartesaghi]

I don't think there is anything wrong with your run. You should be able to confirm that the 5 threads per task are running at 100%. But since you are only using 5 threads per task (or 25 threads for the 5 tilt-series), it will just take a long time to refine the almost 1 million projections in this dataset. In the tutorial, this steps takes ~8 hours using 124 threads per task (or 620 for the 5-tilt series), so using ~25 times less resources will take roughly 25 times longer to run.

[Angeevillor]

I see. Could nextpyp currently capable of GPU acceleration on a single machine? Or is nextpyp only capable of using GPUs on clusters?

[abartesaghi]

Yes, nextPYP can use GPUs even when running in standalone mode, but since the constrained refinement code used in this step of the tutorial is not GPU accelerated, this will not make any difference in terms of running speed.

[tofushan]

The rule of thumb is to use as many resources as possible.

In the tutorial, we use 5 machines, and each of them has 124 cores. Since the five tilt-series are processed in parallel (one for each machine), it takes approximately 8 hours for the reference-based refinement.

Let's say you also use 5 machines, and each of them has 60 CPU cores. Since the five tilt-series can also be processed in parallel, it takes approximately 8 hour * 124 / 80 = 12 hours to run. If this is the case, you should set Threads per task to 60 and check if all the tilt-series are run in parallel. In addition, the estimated time is not always correct, especially for such high workload, because the estimation is simply based on the finished processes.

But it looks like you only have one machine with 60 CPU cores, then there's no way to process 5 tilt-series in parallel (I think it's also for standalone mode), so unfortunately you will need to spend 12 hours * 5 tilt-series = 60 hours. If that is the case, would you mind sharing your computing resources (i.e., number of CPU/GPU machines), so we might consider some other solutions in the future, such as using a combination of CPU and GPU rather than using only one type of machine, etc.

[xingzhe1129]

I am not sure what you said using 5 machines for one refinement job. Do these 5 machines means 5 standalone workstations but both added into nextPYP queue and can one refinement job across 5 workstations? Or do these 5 machines means 5 nodes on a cluster? We have 3 workstations, one has 30 CPUs and 3GPUs, one has 60 CPUs and 8GPUs, and one has 60CPUs and 4GPUs. I think it will great if we can use GPU accelerations or use a combination of CPU and GPU. Thanks.

[tofushan]

I meant to say 5 nodes on a cluster. If you're doing the standalone mode, it requires 60 hours using one workstation with 60 CPU cores. Thank you for providing your resource settings.

[JuhaHuiskonen]

I am having the same issue and not sure how to allocate the resources in the refinement job. Our cluster has as many nodes as we need available and each node has 40 cores. In the tutorial dataset there are 5 tilt-series, so should I ask Threads per task 40 and Max number of threads 200 cores then? Is this the optimal configuration in my case?

[yezhou867]

You are right. If you set the Max number of threads as 0, the jobs will use all the available resources that are allowed for the user. 200 cores also work which allows all 5 jobs to run at the same time, it provides a way to limit the resource occupancy where other things are running.