Platform conformance test hangs on L0 adapter

[lukaszstolarczuk]

Platform fonromance test on L0 adapter (platform-adapter_level_zero) hangs for a long time, causing a timeout. For now, it's disabled on GHA runners (branch adapters).

Hang lasts at least 1,5 hours. Happened on a PC with Intel Arc A750.
Caught when running: #823

[kbenzie]

Which test did it hang on exactly?

[pbalcer]

Just reproduced this:

[==========] Running 33 tests from 8 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from urPlatformCreateWithNativeHandleTest
[ RUN      ] urPlatformCreateWithNativeHandleTest.Success
/.../unified-runtime/test/conformance/platform/fixtures.h:51: Failure
Expected: (count) != (0), actual: 0 vs 0
[  FAILED  ] urPlatformCreateWithNativeHandleTest.Success (24 ms)
[ RUN      ] urPlatformCreateWithNativeHandleTest.InvalidNullPointerPlatform

[kbenzie]

That looks exactly the same as I'm seeing on my 8086K iGPU locally.

[kbenzie]

looks like a deadlock

Thread 1 "test-platform" received signal SIGINT, Interrupt.
0x00007ffff7908cab in sched_yield () at ../sysdeps/unix/syscall-template.S:120
120     ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0  0x00007ffff7908cab in sched_yield () at ../sysdeps/unix/syscall-template.S:120
#1  0x00007ffff7bb299d in urPlatformGet () from /home/benie/Projects/oneapi-src/unified-runtime/build-adapters-r/lib/libur_adapter_level_zero.so.0
#2  0x00007ffff7f841de in urPlatformGet () from /home/benie/Projects/oneapi-src/unified-runtime/build-adapters-r/lib/libur_loader.so.0
#3  0x00005555555776e8 in uur::platform::urPlatformsTest::SetUp() ()
#4  0x0000555555577a34 in uur::platform::urPlatformTest::SetUp() ()
#5  0x00005555555e14a1 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ()
#6  0x00005555555cc6ed in testing::Test::Run() [clone .part.0] ()
#7  0x00005555555cd3e2 in testing::TestInfo::Run() ()
#8  0x00005555555cdc73 in testing::TestSuite::Run() [clone .part.0] ()
#9  0x00005555555d6094 in testing::internal::UnitTestImpl::RunAllTests() ()
#10 0x00005555555cd5c5 in testing::UnitTest::Run() ()
#11 0x0000555555570283 in main ()
(gdb) info threads
  Id   Target Id                                         Frame
* 1    Thread 0x7ffff7e41740 (LWP 36382) "test-platform" 0x00007ffff7908cab in sched_yield () at ../sysdeps/unix/syscall-template.S:120
  2    Thread 0x7ffff6bff640 (LWP 36403) "test-platform" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555559920a0) at ./nptl/futex-internal.c:57
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff6bff640 (LWP 36403))]
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555559920a0) at ./nptl/futex-internal.c:57
57      ./nptl/futex-internal.c: No such file or directory.
(gdb) bt
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555559920a0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5555559920a0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5555559920a0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff7893ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x555555992050, cond=0x555555992078) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x555555992078, mutex=0x555555992050) at ./nptl/pthread_cond_wait.c:627
#5  0x00007ffff7cac77d in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff708f8ff in ?? () from /lib/x86_64-linux-gnu/libze_intel_gpu.so.1
#7  0x00007ffff7894b43 in start_thread (arg=<optimised out>) at ./nptl/pthread_create.c:442
#8  0x00007ffff7926a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

[kbenzie]

I have the L0 driver packages from the intel/compute-runtime 22.35.24055 release installed as that's what is defined in the intel/llvm buildbot/dependency.conf file.

[aarongreig]

Just reproduced this:

[==========] Running 33 tests from 8 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from urPlatformCreateWithNativeHandleTest
[ RUN      ] urPlatformCreateWithNativeHandleTest.Success
/.../unified-runtime/test/conformance/platform/fixtures.h:51: Failure
Expected: (count) != (0), actual: 0 vs 0
[  FAILED  ] urPlatformCreateWithNativeHandleTest.Success (24 ms)
[ RUN      ] urPlatformCreateWithNativeHandleTest.InvalidNullPointerPlatform

The first fail is because of this and us not initializing count here. We should for sure zero init count there but I find the logic in the adapter kind of weird, if this is how we expect NumEntries and pNumPlatforms to interact we should say so in the spec but I'm not sure it is.
(fixing that didn't fix the deadlock)

[aarongreig]

Alright I think at its root this is the whole static platform caching mechanism not reacting well to multiple AdapterGet/AdapterRelease cycles happening in the same application. The global platform list is statically allocated here https://github.com/intel/llvm/blob/sycl/sycl/plugins/unified_runtime/ur/ur.cpp#L25 but then its lifetime is tied to the adapter object's lifetime https://github.com/intel/llvm/blob/sycl/sycl/plugins/unified_runtime/ur/adapters/level_zero/adapter.cpp#L32. The reason this is unique to the platform tests is that they do their adapter management in the regular fixture rather than leaving it to the globally managed environment, which obviously makes sense for the platform testing case.

we should alert @jandres742

[kbenzie]

@jandres742 I've assigned this you're way, I think it slipped under the rug before.

[jandres742]

@jandres742 I've assigned this you're way, I think it slipped under the rug before.

thanks @kbenzie . Taking a look.