Can many indexes share one instance of StandardGpuResources?

[chenqingguo]

Question about GPU.
There are many gpu indexs, including gpuFlat and gpuIVFPQ. And I want them to share one resource.
when I use one StandardGpuResources as a static variable, I got "...CUDA error 29 driver shutting down.",but local variable works.
In my case, each index is an instance, how to manage StandardGpuResources to avoid this error.
thanks.

[mdouze]

Hmmm normally it should be fine to have a StandardGpuResources object per GPU.
Can you share exactly the code that triggers the CUDA error?

[wickedfoo]

The constraint is one StandardGpuResources object per CPU thread. You can use a StandardGpuResources object for multiple indices on the same or multiple GPUs but the constraint is that you cannot run them concurrently from different CPU threads; only a single index + single GPU can be run at one time on a single thread if they share a StandardGpuResources object.

IndexShards and IndexReplicas use multiple threads, so if they are used with GPU indices they require a StandardGpuResources object for each thread / index.

[chenqingguo]

Hmmm normally it should be fine to have a StandardGpuResources object per GPU.
Can you share exactly the code that triggers the CUDA error?

I use docker and only one gpu(2080TI), a resource for multiple indices on a GPU.
Put the resource definition outside the main function, the others are same,which can trigger the CUDA error.

faiss::gpu::StandarGpuResources res;
int main()
{
    int d = 64;                            
    int nb = 100000; 
    int k=10;
    //create data
    faiss::gpu::GpuIndexFlatL2 index_flat(&res, d);
    index_flat.add(nb, xb);
    long *I = new long[k * nq];
    float *D = new float[k * nq];
    index_flat.search(nq, xq, k, D, I);
}

You can use a StandardGpuResources object for multiple indices on the same or multiple GPUs but the constraint is that you cannot run them concurrently from different CPU threads; only a single index + single GPU can be run at one time on a single thread if they share a StandardGpuResources object.

Yes,I have found that. I have 100 GPUIVFPQs, each index has 1 million vectors. The search can not be run on multiple threads.

[wickedfoo]

Yes,I have found that. I have 100 GPUIVFPQs, each index has 1 million vectors. The search can not be run on multiple threads.

Do you have multiple GPUs?
If you run the 100 indices on the same GPU, they will still likely execute sequentially on the GPU itself despite running in different streams if the query is large enough, so it wouldn't help much to run the queries from multiple CPU threads on a single GPU.

Typically with the GPU indices, it only makes sense to go multithreaded CPU either if you have an index on separate GPUs, or if the index is too small or the queries are too small to fully utilize a single GPU.

[chenqingguo]

@wickedfoo ,thanks a lot.

Do you have multiple GPUs?

No,I just have a GPU to deal with 100 million vectors. The number of query is from 20 to 500.

if the index is too small or the queries are too small to fully utilize a single GPU.

How to measure small? Is it determined by GpuResources and GPU memory? Sounds like I don't need to run on multiple threads.

I think some of your instructions should be added to user guideline.