Hardware recommendation?

[info-sic]

Aloha,
on what kind of system is pandora.circle.lu running? It's so much faster than my instance and I want to estimate, which ressources I should make available, when going online...
Currently it's running directly (not dockered) on a VM on VMWare with 4 Cores and 8 GB of RAM.

[Rafiot]

It's running on a machine with 16 cores and 120G of ram, but it definitely doesn't need that much.

The question I have is what is slow? And how many concurrent uploads do you have?

I'd have a look at the CPU usage when a task is being processed to see which processes are running. One thing that might be a problem is the amount of concurrent processes you have for each worker. For example, if you have only one worker of the extractor, every other task has to wait for that worker to be done. You can configure that in the pandora/workers/*.yml

[info-sic]

with the "replicas" parameter?
I checked the procresses while analysing and saw a lot of workers_manager but I can't figure out which process belongs to which kind of worker...
btw, still in pilot-phase I rarely have more than 1 (one) task at a time

[Rafiot]

Yes, the replicas parameter. One task may still trigger a whole bunch of workers (especially if the extractor one is triggered).

There is no easy way to know which worker is which (that's something I need to improve), but if you let me know the type of file that is very slow, I can probably tell you which worker is the problem.

[info-sic]

It's most of the time the preview. I can see a workers_manager eating up 100% of one CPU-Core. But right now I can't reproduce the problem. Anyways, I have other things to fix, like starting pandora from systemd and problems with clamav unable to connect to clamd and poetry run stop not reliably killing everything. I'll get back to you, thanks for your patience and your help!