Document how to use book multiple cores for a single task

[guillaumeeb]

This is to clarify one of the outcome of #181.

Several people have asked how they can use dask-jobqueue to submit multi-threaded tasks. There are two answers currently:

• we can use resource_spec or specific scheduler equivalent kwarg to book more cores than nprocs*nthreads started with dask-worker.
• or we can use resources functionnality, which allows to submit both multi-threaded or single threaded tasks on the same cluster.

This needs to be documented.

@lesteve made up some examples: see #181 (comment).

[djhoese]

Would this also apply to computing dask arrays (client.compute)? For example, if I have a dask array that is the result of many array operations that perform best on a threaded scheduler by themselves, is there an easy way to say that this array and all of its pre-tasks should be computed on a single worker? It's possible the scheduler is smart enough to keep the data on the same node anyway.

[guillaumeeb]

@djhoese you can specify a worker name from client.compute (http://distributed.dask.org/en/latest/api.html#distributed.Client.compute), but I'm not sure resources can be used as a kwarg. Feel free to ask a question in distributed tracker or on Stack Overflow.

[guillaumeeb]

As per the OP, the resources kwarg solution, seems to have some issue yet. See #230 or dask/distributed#1851.

You cannot use adapt() or scale after tasks submission.