interleave::get is "expensive"

[yarikoptic]

Part of troubleshooting of

• pyout might be dragging us behind on large dandisets #1549

running a straightforward download

rm -rf sub-*; DANDI_DEVEL_AGGRESSIVE_RETRY=1 dandi --log-level=INFO download -e refresh -J 1:1 https://api.dandiarchive.org/api/assets/f0da6521-b9c7-4fa5-9405-c48527aebe90/download/

which is a single file/thread download leads to dandi CLI being 10-20% CPU and py-spy showing

so sampling profiler finds itself most often in get (interleave/__init__.py). There are multiple gets in that file but this particular Interleaver.get https://github.com/jwodder/interleave/blob/master/src/interleave/__init__.py#L348 is the primary suspect due to internal while True loop.

@jwodder could you please see if code there potentially be made more efficient?

[jwodder]

@yarikoptic I can't think of a way to make it more efficient. Note that the while True loop should largely be spending its time waiting for queue.SimpleQueue.get().