BUG) wrong gpu assignment

[YoojLee]

Bug description

The latest version of code on the main branch seems to have a bug regarding a GPU assignment in the nearest neighbor search (Faiss) process at the inference.

I tested on two environments:

• Python 3.6.9 and faiss-gpu==1.5.3
• Python 3.8.10, Conda 4.8.3 and faiss-gpu==1.7.3.

Whatever GPU I assign for the code, at the inference (nearest neighbor search for anomaly score), GPU 0 unexpectedly participated in the search. This might lead to computational inefficiency.

Reproduction of the failure case

I reproduced the error as the below figure presents (I assigned GPU 1 for the code, but at the inference, GPU 0 is unexpectedly involved in the process):

→ When computing features from training images, GPU 0 is idle, and no data is assigned to the device. It seems like GPU 1 is the only participant in the anomaly detection procedure.

→ However, as an inference begins, data is assigned to GPU 0, and the nearest neighbor is computed on GPU 0 (memory assigned and utilization is non-zero).

This bug is caused by line 46 from common.py, as described below:

class FaissNN(object):
    ...
    def _create_index(self, dimension):
        if self.on_gpu:
            return faiss.GpuIndexFlatL2(
                faiss.StandardGpuResources(), dimension, faiss.GpuIndexFlatConfig()
            )
        return faiss.IndexFlatL2(dimension)

Suggestion

In the original code, you use the default configuration for FaissGPUIndexFlatL2, where the default device for GPU is set to 0. I modded the code to assign the expected device to the configuration (details are provided in the pull request I made)

After the fix:

→ The issue has been solved as above

→ I confirmed there is no adverse effect on the performance as well.

[onestardao]

great catch — this looks like a textbook case of what we classified as Problem Map No.9: untracked context handoff.
basically when GPU context silently shifts between training and inference, especially under mixed device defaults or implicit reallocation.

we’ve seen this creep into multi-GPU deployments and containerized setups too — usually hard to trace unless you’re logging every device move.

if you're curious, we’ve built a full map of these failure patterns (16+ categories), with matching symbolic fixes.
open-source, MIT-licensed, and it’s been useful for folks trying to kill debug time in pipeline bugs like this.

happy to share if you’re interested — just let me know.