Build system dependency issue with syscalls

[andrewboie]

Describe the bug
I'm seeing a problem where make clean; make -j20 is run and the dependencies look messed up. This was found in code review for #23429 but is reproducible without that PR:

apboie@apboie-mobl4:~/projects/zephyr3/zephyr/tests/kernel/mem_protect/userspace/out master /home/apboie/projects/zephyr3/zephyr
$ mj
[  0%] Preparing syscall dependency handling
[  1%] Preparing syscall dependency handling
[  1%] Preparing syscall dependency handling



[  2%] Generating misc/generated/syscalls_subdirs.trigger
[  3%] Generating misc/generated/syscalls_subdirs.trigger
[  4%] Generating misc/generated/syscalls_subdirs.trigger
[  4%] Generating misc/generated/syscalls.json, misc/generated/subsystems.json
[  5%] Generating misc/generated/syscalls.json, misc/generated/subsystems.json

In addition, occasionally when doing this the build fails:

Traceback (most recent call last):
  File "/home/apboie/projects/zephyr3/zephyr/scripts/subfolder_list.py", line 80, in <module>
    main()
  File "/home/apboie/projects/zephyr3/zephyr/scripts/subfolder_list.py", line 56, in main
    os.symlink(directory, symlink)
FileExistsError: [Errno 17] File exists: '/home/apboie/projects/zephyr3/zephyr/include/dt-bindings/interrupt-controller' -> '/home/apboie/projects/zephyr3/zephyr/tests/kernel/mem_protect/userspace/out/zephyr/misc/generated/syscalls_links/_home_apboie_projects_zephyr3_zephyr_include_dt-bindings_interrupt-controller'
[  5%] Generating include/generated/driver-validation.h
[  6%] Generating include/generated/kobj-types-enum.h, include/generated/otype-to-str.h
zephyr/CMakeFiles/syscall_list_h_target.dir/build.make:166: recipe for target 'zephyr/misc/generated/syscalls_subdirs.trigger' failed
make[2]: *** [zephyr/misc/generated/syscalls_subdirs.trigger] Error 1
make[2]: *** Deleting file 'zephyr/misc/generated/syscalls_subdirs.trigger'
CMakeFiles/Makefile2:797: recipe for target 'zephyr/CMakeFiles/syscall_list_h_target.dir/all' failed
make[1]: *** [zephyr/CMakeFiles/syscall_list_h_target.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[  6%] Built target driver_validation_h_target
[  6%] Built target kobj_types_h_target
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

This seems a race condition which only reproduces sometimes and depends on the host workstation characteristics and the -j parameter to make.

To reproduce:

cd tests/kernel/mem_protect/userspace
mkdir out
cd out
cmake -DBOARD=qemu_x86 ..
make -j20
make clean
make -j20

[andrewboie]

Through bisection this PR seems to be the culprit: #23183

[andrewboie]

@xodus7 FYI

[andrewboie]

This seems to "fix" it, but I don't know if this is the right thing to do as we don't express dependencies on the subsystem json any more.

Maybe the best thing to do is just combine syscalls_json and subsys_json into one file

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 82407c7284..51e55f6df4 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -586,7 +586,6 @@ endforeach()
 add_custom_command(
   OUTPUT
   ${syscalls_json}
-  ${subsys_json}
   COMMAND
   ${PYTHON_EXECUTABLE}
   ${ZEPHYR_BASE}/scripts/parse_syscalls.py
@@ -634,7 +633,6 @@ add_custom_command(
   $<$<BOOL:${CMAKE_VERBOSE_MAKEFILE}>:--verbose>
   DEPENDS
   ${ZEPHYR_BASE}/scripts/gen_kobject_list.py
-  ${subsys_json}
   WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
   )
 add_custom_target(${DRIVER_VALIDATION_H_TARGET} DEPENDS ${DRV_VALIDATION})
@@ -954,7 +952,6 @@ if(CONFIG_USERSPACE)
     $<$<BOOL:${CMAKE_VERBOSE_MAKEFILE}>:--verbose>
     DEPENDS
     ${ZEPHYR_PREBUILT_EXECUTABLE}
-    ${subsys_json}
     WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
     )
   add_custom_target(obj_list DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${OBJ_LIST})
diff --git a/cmake/kobj.cmake b/cmake/kobj.cmake
index f54e148a60..1b751cadb9 100644
--- a/cmake/kobj.cmake
+++ b/cmake/kobj.cmake
@@ -25,7 +25,6 @@ function(gen_kobj gen_dir_out)
     $<$<BOOL:${CMAKE_VERBOSE_MAKEFILE}>:--verbose>
     DEPENDS
     $ENV{ZEPHYR_BASE}/scripts/gen_kobject_list.py
-    ${subsys_json}
     WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
     )
   add_custom_target(${KOBJ_TYPES_H_TARGET} DEPENDS ${KOBJ_TYPES} ${KOBJ_OTYPE})

[xodus7]

Interesting, sorry that I didn't notice this issue in my original PR.