KillWait default in slurm slurm

[ilya-da]

at this moment deault slurm 30 sec KillWait used, but this may be insufficient for large parallel jobs to gracefully terminate.

to remind it define period in second between try to kill (for example because of timeout reached) by SIGTERM until SIGKILL used.
SIGKILL may cause

Deallocating GPU memory and closing GPU contexts take time, especially when multiple GPUs are involved, apart will most likely disk cleanup involved.
Allowing longer grace periods for job termination can help prevent data loss and ensure proper cleanup of resources, especially for large-scale applications that may require more time to save state and release resources.

[ilya-da]

#1318 possible solution