Assorted userspace cleanups #23429

andrewboie · 2020-03-12T19:50:18Z

Poorly named structs _k_object, _k_thread_stack_element, _k_object_assignment renamed to use z_ prefix, all are kernel-private
The data member of struct z_object is now a union making the semantics much clearer
Generation of privilege mode stacks is no longer a separate pass, logic merged into gen_kobject_list.py. this should speed up build times on ARM platforms.
elf_helper.py merged back into gen_kobject_list.py

zephyrbot · 2020-03-12T20:00:11Z

All checks are passing now.

checkpatch (informational only, not a failure)

-:932: WARNING:NEW_TYPEDEFS: do not add new typedefs
#932: FILE: include/sys/arch_interface.h:44:
+typedef struct z_thread_stack_element k_thread_stack_t;

- total: 0 errors, 1 warnings, 1871 lines checked

NOTE: For some of the reported defects, checkpatch may be able to
      mechanically convert to the typical style using --fix or --fix-inplace.

Your patch has style problems, please review.

NOTE: Ignored message types: AVOID_EXTERNS BRACES CONFIG_EXPERIMENTAL CONST_STRUCT DATE_TIME FILE_PATH_CHANGES MINMAX NETWORKING_BLOCK_COMMENT_STYLE PRINTK_WITHOUT_KERN_LEVEL SPLIT_STRING VOLATILE

NOTE: If any of the errors are false positives, please report
      them to the maintainers.

Tip: The bot edits this comment instead of posting a new one, so you can check the comment's history to see earlier messages.

andrewboie · 2020-03-13T17:31:14Z

This seems to have broken make clean; make, investigating...

Traceback (most recent call last):
  File "/home/apboie/projects/zephyr2/zephyr/scripts/subfolder_list.py", line 80, in <module>
    main()
  File "/home/apboie/projects/zephyr2/zephyr/scripts/subfolder_list.py", line 44, in main
    os.symlink(directory, symlink)
FileExistsError: [Errno 17] File exists: '/home/apboie/projects/zephyr2/zephyr/include' -> '/home/apboie/projects/zephyr2/zephyr/tests/kernel/mem_protect/userspace/out/zephyr/misc/generated/syscalls_links/_home_apboie_projects_zephyr2_zephyr_include'
Traceback (most recent call last):
  File "/home/apboie/projects/zephyr2/zephyr/scripts/subfolder_list.py", line 80, in <module>
    main()
  File "/home/apboie/projects/zephyr2/zephyr/scripts/subfolder_list.py", line 44, in main
    os.symlink(directory, symlink)
FileExistsError: [Errno 17] File exists: '/home/apboie/projects/zephyr2/zephyr/include' -> '/home/apboie/projects/zephyr2/zephyr/tests/kernel/mem_protect/userspace/out/zephyr/misc/generated/syscalls_links/_home_apboie_projects_zephyr2_zephyr_include'
zephyr/CMakeFiles/kobj_types_h_target.dir/build.make:168: recipe for target 'zephyr/misc/generated/syscalls_subdirs.trigger' failed
make[2]: *** [zephyr/misc/generated/syscalls_subdirs.trigger] Error 1
make[2]: *** Deleting file 'zephyr/misc/generated/syscalls_subdirs.trigger'
CMakeFiles/Makefile2:733: recipe for target 'zephyr/CMakeFiles/kobj_types_h_target.dir/all' failed
make[1]: *** [zephyr/CMakeFiles/kobj_types_h_target.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[  5%] Generating include/generated/syscall_dispatch.c, include/generated/syscall_list.h
zephyr/CMakeFiles/driver_validation_h_target.dir/build.make:164: recipe for target 'zephyr/misc/generated/syscalls_subdirs.trigger' failed
make[2]: *** [zephyr/misc/generated/syscalls_subdirs.trigger] Error 1
CMakeFiles/Makefile2:765: recipe for target 'zephyr/CMakeFiles/driver_validation_h_target.dir/all' failed
make[1]: *** [zephyr/CMakeFiles/driver_validation_h_target.dir/all] Error 2
[  5%] Built target syscall_list_h_target
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

andrewboie · 2020-03-13T17:37:51Z

@SebastianBoe I'm not sure what part of my CMakeLists.txt changes broke this, but I think there is an underlying bug -- even in a vanilla Zephyr tree, when I run "make clean" there's still tons of generated stuff left in build/zephyr/include/generated

SebastianBoe · 2020-03-16T15:43:45Z

I was not able to reproduce the issue of make clean not working.

But I see that this happens:

[  1%] Preparing syscall dependency handling
[  1%] Preparing syscall dependency handling
[  2%] Preparing syscall dependency handling



[  3%] Generating misc/generated/syscalls_subdirs.trigger
[  4%] Generating misc/generated/syscalls_subdirs.trigger
[  5%] Generating misc/generated/syscalls_subdirs.trigger
[  6%] Generating misc/generated/syscalls.json, misc/generated/subsystems.json
[  7%] Generating misc/generated/syscalls.json, misc/generated/subsystems.json
[  7%] Generating misc/generated/syscalls.json, misc/generated/subsystems.json

Which usually means that commands are depending on a file, instead of a target that wraps the file.

"You can’t just depend on gen1 instead of gen1-wrapper, because it may end up being built multiple times!"

https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-targets-and-files-and-custom-commands/

SebastianBoe

Dependencies need to be fixed.

andrewboie · 2020-03-16T17:11:56Z

I was not able to reproduce the issue of make clean not working.

But I see that this happens:
[  1%] Preparing syscall dependency handling
[  1%] Preparing syscall dependency handling
[  2%] Preparing syscall dependency handling



[  3%] Generating misc/generated/syscalls_subdirs.trigger
[  4%] Generating misc/generated/syscalls_subdirs.trigger
[  5%] Generating misc/generated/syscalls_subdirs.trigger
[  6%] Generating misc/generated/syscalls.json, misc/generated/subsystems.json
[  7%] Generating misc/generated/syscalls.json, misc/generated/subsystems.json
[  7%] Generating misc/generated/syscalls.json, misc/generated/subsystems.json
Which usually means that commands are depending on a file, instead of a target that wraps the file.

"You can’t just depend on gen1 instead of gen1-wrapper, because it may end up being built multiple times!"

samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-targets-and-files-and-custom-commands

@SebastianBoe I can reproduce this without this PR.

For example, in master branch, I built, 'make clean', and 'make' again:

apboie@apboie-mobl4:~/projects/zephyr3/zephyr/tests/kernel/mem_protect/userspace/out master /home/apboie/projects/zephyr3/zephyr
$ mj
[  0%] Preparing syscall dependency handling
[  1%] Preparing syscall dependency handling
[  1%] Preparing syscall dependency handling



[  2%] Generating misc/generated/syscalls_subdirs.trigger
[  3%] Generating misc/generated/syscalls_subdirs.trigger
[  4%] Generating misc/generated/syscalls_subdirs.trigger
[  4%] Generating misc/generated/syscalls.json, misc/generated/subsystems.json
[  5%] Generating misc/generated/syscalls.json, misc/generated/subsystems.json
Traceback (most recent call last):
  File "/home/apboie/projects/zephyr3/zephyr/scripts/subfolder_list.py", line 80, in <module>
    main()
  File "/home/apboie/projects/zephyr3/zephyr/scripts/subfolder_list.py", line 56, in main
    os.symlink(directory, symlink)
FileExistsError: [Errno 17] File exists: '/home/apboie/projects/zephyr3/zephyr/include/dt-bindings/interrupt-controller' -> '/home/apboie/projects/zephyr3/zephyr/tests/kernel/mem_protect/userspace/out/zephyr/misc/generated/syscalls_links/_home_apboie_projects_zephyr3_zephyr_include_dt-bindings_interrupt-controller'
[  5%] Generating include/generated/driver-validation.h
[  6%] Generating include/generated/kobj-types-enum.h, include/generated/otype-to-str.h
zephyr/CMakeFiles/syscall_list_h_target.dir/build.make:166: recipe for target 'zephyr/misc/generated/syscalls_subdirs.trigger' failed
make[2]: *** [zephyr/misc/generated/syscalls_subdirs.trigger] Error 1
make[2]: *** Deleting file 'zephyr/misc/generated/syscalls_subdirs.trigger'
CMakeFiles/Makefile2:797: recipe for target 'zephyr/CMakeFiles/syscall_list_h_target.dir/all' failed
make[1]: *** [zephyr/CMakeFiles/syscall_list_h_target.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[  6%] Built target driver_validation_h_target
[  6%] Built target kobj_types_h_target
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

I think it is a race condition on whether the failure occurs, which is why I don't always see it.

Please reconsider your -1 as this issue is not related to this PR.

andrewboie · 2020-03-16T19:24:39Z

I opened #23504 as the problems seen are unrelated to this PR.

andrewboie · 2020-03-16T19:34:55Z

@SebastianBoe I've root-caused these problems to #23183 it's not this patch.
Edit: I have a fix #23505

SebastianBoe

Removing NACK

Rather than stuffing various values in a uintptr_t based on type using casts, use a union for this instead. No functional difference, but the semantics of the data member are now much clearer to the casual observer since it is now formally defined by this union. Signed-off-by: Andrew Boie <[email protected]>

Private data type, prefix with z_. Signed-off-by: Andrew Boie <[email protected]>

Private structure, rename to z_object_assignment Signed-off-by: Andrew Boie <[email protected]>

Private type, internal to the kernel, not directly associated with any k_object_* APIs. Is the return value of z_object_find(). Rename to struct z_object. Signed-off-by: Andrew Boie <[email protected]>

These were renamed to z_ prefix some time ago, but not updated in these places. Signed-off-by: Andrew Boie <[email protected]>

This never needed to be put in a separate gperf table. Privilege mode stacks can be generated by the main gen_kobject_list.py logic, which we do here. Signed-off-by: Andrew Boie <[email protected]>

No need for this to be separated out any more. Minimal changes made to get it to still work. Signed-off-by: Andrew Boie <[email protected]>

Un-do the changes made to the code when this was separated out to elf_helper.py. Signed-off-by: Andrew Boie <[email protected]>

andrewboie requested review from MaureenHelm, SebastianBoe, agross-oss, andyross, carlescufi, carlocaione, galak, ioannisg, nashif, stephanosio and wentongwu as code owners March 12, 2020 19:50

zephyrbot added area: ARM ARM (32-bit) Architecture area: API Changes to public APIs area: Tests Issues related to a particular existing or missing test area: Kernel area: Documentation area: Build System labels Mar 12, 2020

nashif mentioned this pull request Mar 12, 2020

merge gen_kobject_list.py and gen_priv_stacks.py #19588

Closed

andrewboie force-pushed the userspace-cleanups branch 3 times, most recently from a7561ab to cda7887 Compare March 12, 2020 20:45

nashif approved these changes Mar 13, 2020

View reviewed changes

andrewboie force-pushed the userspace-cleanups branch 3 times, most recently from 0814a45 to a46d3f6 Compare March 13, 2020 17:29

andrewboie mentioned this pull request Mar 13, 2020

"make clean" doesn't clean a lot of generated files #23449

Closed

andrewboie requested a review from ceolin March 13, 2020 21:31

andrewboie requested a review from dcpleung March 13, 2020 21:31

dcpleung approved these changes Mar 13, 2020

View reviewed changes

andyross approved these changes Mar 14, 2020

View reviewed changes

SebastianBoe requested changes Mar 16, 2020

View reviewed changes

andrewboie mentioned this pull request Mar 16, 2020

Build system dependency issue with syscalls #23504

Closed

SebastianBoe approved these changes Mar 17, 2020

View reviewed changes

andrewboie force-pushed the userspace-cleanups branch from a46d3f6 to 383c098 Compare March 17, 2020 16:09

Andrew Boie added 8 commits March 17, 2020 09:50

kernel: rename _k_thread_stack_element

d05d5c3

Private data type, prefix with z_. Signed-off-by: Andrew Boie <[email protected]>

kernel: rename _k_object_assignment

7fb65bf

Private structure, rename to z_object_assignment Signed-off-by: Andrew Boie <[email protected]>

kernel: rename struct _k_object

d6131ba

Private type, internal to the kernel, not directly associated with any k_object_* APIs. Is the return value of z_object_find(). Rename to struct z_object. Signed-off-by: Andrew Boie <[email protected]>

doc: fix some missed renames

1c41602

These were renamed to z_ prefix some time ago, but not updated in these places. Signed-off-by: Andrew Boie <[email protected]>

kernel: delete separate logic for priv stacks

932bba5

This never needed to be put in a separate gperf table. Privilege mode stacks can be generated by the main gen_kobject_list.py logic, which we do here. Signed-off-by: Andrew Boie <[email protected]>

scripts: merge elf_helper into gen_kobject_list

2f3d64c

No need for this to be separated out any more. Minimal changes made to get it to still work. Signed-off-by: Andrew Boie <[email protected]>

scripts: gen_kobject_list.py cleanup

3c52325

Un-do the changes made to the code when this was separated out to elf_helper.py. Signed-off-by: Andrew Boie <[email protected]>

andrewboie force-pushed the userspace-cleanups branch from 383c098 to 3c52325 Compare March 17, 2020 16:58

andrewboie merged commit b42fe9c into zephyrproject-rtos:master Mar 17, 2020

andrewboie deleted the userspace-cleanups branch March 17, 2020 18:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Assorted userspace cleanups #23429

Assorted userspace cleanups #23429

Uh oh!

andrewboie commented Mar 12, 2020 •

edited

Loading

Uh oh!

zephyrbot commented Mar 12, 2020 •

edited

Loading

Uh oh!

andrewboie commented Mar 13, 2020

Uh oh!

andrewboie commented Mar 13, 2020

Uh oh!

SebastianBoe commented Mar 16, 2020

Uh oh!

SebastianBoe left a comment

Uh oh!

andrewboie commented Mar 16, 2020

Uh oh!

andrewboie commented Mar 16, 2020

Uh oh!

andrewboie commented Mar 16, 2020 •

edited

Loading

Uh oh!

SebastianBoe left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Assorted userspace cleanups #23429

Assorted userspace cleanups #23429

Uh oh!

Conversation

andrewboie commented Mar 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zephyrbot commented Mar 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

checkpatch (informational only, not a failure)

Uh oh!

andrewboie commented Mar 13, 2020

Uh oh!

andrewboie commented Mar 13, 2020

Uh oh!

SebastianBoe commented Mar 16, 2020

Uh oh!

SebastianBoe left a comment

Choose a reason for hiding this comment

Uh oh!

andrewboie commented Mar 16, 2020

Uh oh!

andrewboie commented Mar 16, 2020

Uh oh!

andrewboie commented Mar 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SebastianBoe left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

andrewboie commented Mar 12, 2020 •

edited

Loading

zephyrbot commented Mar 12, 2020 •

edited

Loading

andrewboie commented Mar 16, 2020 •

edited

Loading