`dpctl.program._program.SyclProgramCompilationError` for some kernels for cpu target (latest updates)

[fcharras]

With latest updates (>=0.21.2 I think) some of my kernels can't compile anymore when the device is "cpu":

sklearn_numba_dpex/kmeans/engine.py:252: in get_labels
    labels, _ = self._get_labels_inertia(X, sample_weight, with_inertia=False)
sklearn_numba_dpex/kmeans/engine.py:266: in _get_labels_inertia
    assignments_idx, inertia = get_labels_inertia(
sklearn_numba_dpex/kmeans/drivers.py:663: in get_labels_inertia
    label_assignment_fixed_window_kernel(
/opt/venv/lib/python3.10/site-packages/numba_dpex/core/kernel_interface/dispatcher.py:467: in __call__
    kernel_bundle = dpctl_prog.create_program_from_spirv(
dpctl/program/_program.pyx:274: in dpctl.program._program.create_program_from_spirv
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   dpctl.program._program.SyclProgramCompilationError

dpctl/program/_program.pyx:311: SyclProgramCompilationError

minimal reproducer is hard to achieve. The surfaced error does not give information unfortunately.

[fcharras]

I think I can track down this one to #1103 , some kernels apparently would still require atomic emulation to be compiled correctly for CPU.

Is this a bug or a feature that atomic on CPU does not consistently work anymore ? Compiling and running all kernels on CPU is super attractive because it enables access to a cheap cpu-only CI, I really wish we can have that.

[ZzEeKkAa]

I was able to create small reproducer for this issue:

import dpnp as np

import numba_dpex as ndpx


@ndpx.kernel
def ker(a):
    private = ndpx.private.array(shape=1, dtype=a.dtype)
    ndpx.atomic.add(private, 0, 1) # does not work
    # ndpx.atomic.add(a, 0, 1) # does work


def main():
    a = np.arange(0, 1)

    ker[ndpx.Range(a.size)](a)


if __name__ == "__main__":
    main()

Sycl compilation error appears only if atomic is being used on private memory (is it the right way of using atomics?). It is worth to mention that atomic works on the array allocated on device. So it may be not related to lack of atomic support for particular hardware.

Issue is only reproducible on opencl:cpu. opencl:gpu works without any issues.

[adarshyoga]

Atomics on private memory does not make sense! It should be disallowed once the new changes to atomics come in.

We are in the middle of making changes to atomic operations to make them similar to how they are handled in sycl. These changes include a new AtomicRef type which holds a reference to the memory location that needs to be handled atomically.
With this new type we perform a check to to ensure the reference is of type USMNDArray. This check will prevent the user from creating AtomicRef objects out of private memory references.

[fcharras]

Thank you @ZzEeKkAa I think you spot the real issue there. Definitely atomics on private memory does not make sense... but I used it sometime as a (desperate) workaround for #1106 . In this case, indeed the issue is not about removal of atomic emulation.

[fcharras]

I'm finding other workarounds for #1106 and I can drop the hackish atomic on private memories and move on, since I can now run successfully all tests on CPU with the latest commit on main (following #1158). This issue is no longer needed, further investigation can take place in #1106 thread. TY all.