[RFC][PoC] Zephyr PreProcessor (ZPP) framework #82990

pdgendt · 2024-12-14T13:53:25Z

Introduction

This is a proof-of-concept, namely the Zephyr Pre-Processor (ZPP) framework, an alternative to the syscall parsing for gathering function and struct information.

Problem description

The https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/build/parse_syscalls.py script has a few drawbacks that this PR tries to improve upon.

It's currently (ab)used to find __subsystem and __net_socket "tags" but this is a hidden implementation detail
It operates on header/source files before the C pre-processer so we can't rely on macro expansion
It generates more boiler plate than is needed

Proposed change

Run the C pre-processor, in the pre-build stage, on files that are part of the build and have the -DZPP compile definition (opt-in), and parse the output for annotated functions or structs.
Gather the information in a formatted (json) file that can be fed into other build steps.

Detailed RFC

In the aforementioned C pre-processor run, the compile definition -D__ZPP__ is added which enables the C23 annotations (not part of the regular build), for example:

#ifdef __ZPP__
#define __zpp(x) [[zephyr::x]]
#else
#define __zpp(...)
#endif

#define __syscall __zpp(func("syscall", __FILE__, __LINE__))
#define __subsystem __zpp(struct("__subsystem"))

The newly introduced scripts/zpp.py scans for these annotations and can output the structured data needed for the next build steps.

[
  {
    "attr": "func",
    "args": [
      "syscall",
      "WEST_TOPDIR/zephyr/include/zephyr/kernel.h",
      537
    ],
    "data": {
      "type": "int",
      "name": "k_thread_join",
      "args": "struct k_thread *thread, k_timeout_t timeout"
    }
  },
  {
    "attr": "struct",
    "args": [
      "__subsystem"
    ],
    "data": {
      "name": "gpio_driver_api"
    }
  }
]

Proposed change (Detailed)

Convert the __subsystem tags with ZPP (part of the PR already)
Convert the __net_socket tags with ZPP
Convert the __syscall tags with ZPP
Introduce __iterable_section_rom tags for structs to allow creating linker sections from source files, without the need for additional linker scripts (cleanup all common-rom.ld and common-rom.cmake files)

Dependencies

Affects downstream usage of the syscall mechanism.

Concerns and Unresolved Questions

Portability for different toolchains
Opt-in vs opt-out
~~Generated headers need include guards when running ZPP (part of the PR already)~~

Alternatives

Don't do this

henrikbrixandersen · 2024-12-16T10:34:26Z

That's a very interesting idea. I like it, my only concern is "Portability for different toolchains" - do you any further data points on this?

scripts/zpp.py

pdgendt · 2024-12-16T10:41:08Z

That's a very interesting idea. I like it, my only concern is "Portability for different toolchains" - do you any further data points on this?

It comes down to the C compiler support for the following options:

-E -P to only do the preprocessing step (not an issue, I think)
-M -MG print dependency list, as well as missing include headers (relative paths)

Documentation is scarce, I based the implementation on GCC options
Support for clang should be covered too.

scripts/zpp.py

Copilot reviewed 2 out of 18 changed files in this pull request and generated no comments.

Files not reviewed (16)

CMakeLists.txt: Language not supported
cmake/linker/ld/target.cmake: Language not supported
cmake/linker/linker_script_common.cmake: Language not supported
cmake/linker_script/common/common-rom.cmake: Language not supported
cmake/zpp.cmake: Language not supported
include/zephyr/linker/common-rom.ld: Language not supported
include/zephyr/linker/common-rom/common-rom-kernel-devices.ld: Language not supported
include/zephyr/linker/common-rom/common-rom-net.ld: Language not supported
include/zephyr/net/coap_service.h: Language not supported
include/zephyr/net/dns_sd.h: Language not supported
include/zephyr/net/http/service.h: Language not supported
include/zephyr/net/net_mgmt.h: Language not supported
include/zephyr/net/socket.h: Language not supported
include/zephyr/net/socket_service.h: Language not supported
include/zephyr/toolchain/common.h: Language not supported
subsys/net/l2/ppp/ppp_internal.h: Language not supported

pdgendt · 2024-12-16T13:01:23Z

Copilot reviewed 2 out of 18 changed files in this pull request and generated no comments.

Thanks Copilot

gmarull · 2024-12-17T09:03:15Z

include/zephyr/toolchain/common.h

+#ifndef __ZPP__
 #define __syscall static inline
 #define __syscall_always_inline static inline __attribute__((always_inline))
 #else
+#define __syscall __zpp(func("syscall", __FILE__, __LINE__))
+#define __syscall_always_inline __zpp(func("syscall", __FILE__, __LINE__))


question: both expand to the same?

The __syscall part hasn't been refactored yet, and this is mostly to demonstrate part of the output.

But I think the current syscall parsing also behaves the same, they end up different when compiling only.

gmarull · 2024-12-17T09:03:51Z

include/zephyr/toolchain/common.h

 /* Indicates this is a driver subsystem */
-#define __subsystem
+#define __subsystem __zpp(struct("__subsystem"))

 /* Indicates this is a network socket object */
-#define __net_socket
+#define __net_socket __zpp(struct("__net_socket"))


for future: these should likely be moved somewhere else

gmarull · 2024-12-17T09:05:08Z

cmake/zpp.cmake

+# Generate compile_commands.json in the output directory
+set(CMAKE_EXPORT_COMPILE_COMMANDS TRUE CACHE BOOL
+    "Export CMake compile commands. Used by zpp.py script"
+    FORCE
+)


this is already set in cmake/modules/kernel.cmake

I copied it from there, it should be merged somewhere indeed, but for the PoC it was easier to have it here with less chance of getting conflicts.

bjarki-andreasen · 2024-12-17T17:27:04Z

Overall this seems nice, making the compiler do some of the parsing work we do by "hand" now, just hope these attributes are well supported by all the compilers we support :)

pdgendt · 2024-12-23T11:32:26Z

That's a very interesting idea. I like it, my only concern is "Portability for different toolchains" - do you any further data points on this?

It comes down to the C compiler support for the following options:

-E -P to only do the preprocessing step (not an issue, I think)

-M -MG print dependency list, as well as missing include headers (relative paths)

Documentation is scarce, I based the implementation on GCC options Support for clang should be covered too.

@RobinKastberg does IAR support the C compiler flags -E -P -M -MG?

RobinKastberg · 2025-01-08T11:35:25Z

@pdgendt Without looking too closely I think this should be no problem (It seems to map to the --preprocess and --dependencies flag)

RobinKastberg · 2025-01-08T11:59:41Z

@pdgendt C23 annotations might be harder for us right now, but that should be possible to work around?

pdgendt · 2025-01-08T12:21:16Z

@pdgendt C23 annotations might be harder for us right now, but that should be possible to work around?

The C23 annotations aren't used after the pre-processor step currently, they are simply to make it parse-able.

Add a script to act as a pre-processor to capture functions and structs that require special attention. Signed-off-by: Pieter De Gendt <[email protected]>

Update some directives to add a C23 annotation instead. Signed-off-by: Pieter De Gendt <[email protected]>

Create a CMake target to build the annotations together with a depfile. Signed-off-by: Pieter De Gendt <[email protected]>

Rework gen_iter_sections.py to use the annotations json file generated by ZPP. Signed-off-by: Pieter De Gendt <[email protected]>

TO BE REMOVED For testing enable the Zephyr PreProcessor on all source files. Signed-off-by: Pieter De Gendt <[email protected]>

Add a new tag for generating linker sections, namely __iterable_section_rom and rename the device API logic to a more generic one. Signed-off-by: Pieter De Gendt <[email protected]>

Add __iterable_section_rom to network struct types and remove ld/cmake entries. Signed-off-by: Pieter De Gendt <[email protected]>

pdgendt added the RFC Request For Comments: want input from the community label Dec 14, 2024

pdgendt requested review from kartben, henrikbrixandersen, tejlmand, gmarull and bjarki-andreasen December 14, 2024 13:53

pdgendt force-pushed the poc-zpp branch 11 times, most recently from 5af477d to 185bcac Compare December 16, 2024 09:17

henrikbrixandersen added the area: Build System label Dec 16, 2024

kartben reviewed Dec 16, 2024

View reviewed changes

scripts/zpp.py Outdated Show resolved Hide resolved

pdgendt force-pushed the poc-zpp branch from 185bcac to 5e78385 Compare December 16, 2024 11:04

gmarull reviewed Dec 16, 2024

View reviewed changes

scripts/zpp.py Outdated Show resolved Hide resolved

pdgendt force-pushed the poc-zpp branch from 5e78385 to b6c3e4f Compare December 16, 2024 12:41

kartben requested a review from Copilot December 16, 2024 12:58

Copilot AI reviewed Dec 16, 2024

View reviewed changes

pdgendt force-pushed the poc-zpp branch 2 times, most recently from 575cd12 to f0d72d0 Compare December 16, 2024 16:08

gmarull reviewed Dec 17, 2024

View reviewed changes

pdgendt force-pushed the poc-zpp branch from f0d72d0 to 4118a92 Compare December 17, 2024 15:17

pillo79 mentioned this pull request Jan 8, 2025

tests: llext-edk: get the Zephyr SDK path from build_info.yml #83661

Merged

pdgendt added 5 commits January 10, 2025 10:57

scripts: Introduce Zephyr PreProcessor (ZPP)

73d5e74

Add a script to act as a pre-processor to capture functions and structs that require special attention. Signed-off-by: Pieter De Gendt <[email protected]>

include: zephyr: toolchain: Add ZPP directives

ac30696

Update some directives to add a C23 annotation instead. Signed-off-by: Pieter De Gendt <[email protected]>

cmake: Add zpp target

a61711e

Create a CMake target to build the annotations together with a depfile. Signed-off-by: Pieter De Gendt <[email protected]>

scripts: build: Use annotations file for gen_iter_sections.py

1de5270

Rework gen_iter_sections.py to use the annotations json file generated by ZPP. Signed-off-by: Pieter De Gendt <[email protected]>

TMP: Globally enable ZPP

5fb2e12

TO BE REMOVED For testing enable the Zephyr PreProcessor on all source files. Signed-off-by: Pieter De Gendt <[email protected]>

pdgendt force-pushed the poc-zpp branch from 4118a92 to bc86c41 Compare January 10, 2025 10:41

pdgendt added 2 commits January 10, 2025 12:07

cmake: Make device API linker logic more generic

067eb43

Add a new tag for generating linker sections, namely __iterable_section_rom and rename the device API logic to a more generic one. Signed-off-by: Pieter De Gendt <[email protected]>

net: Add tags to network iterable ROM sections

6c410db

Add __iterable_section_rom to network struct types and remove ld/cmake entries. Signed-off-by: Pieter De Gendt <[email protected]>

pdgendt force-pushed the poc-zpp branch from bc86c41 to 6c410db Compare January 10, 2025 11:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC][PoC] Zephyr PreProcessor (ZPP) framework #82990

[RFC][PoC] Zephyr PreProcessor (ZPP) framework #82990

pdgendt commented Dec 14, 2024 •

edited

Loading

henrikbrixandersen commented Dec 16, 2024

pdgendt commented Dec 16, 2024 •

edited

Loading

pdgendt commented Dec 16, 2024

gmarull Dec 17, 2024

pdgendt Dec 17, 2024

gmarull Dec 17, 2024

pdgendt Dec 17, 2024

gmarull Dec 17, 2024

pdgendt Dec 17, 2024 •

edited

Loading

bjarki-andreasen commented Dec 17, 2024 •

edited

Loading

pdgendt commented Dec 23, 2024

RobinKastberg commented Jan 8, 2025

RobinKastberg commented Jan 8, 2025

pdgendt commented Jan 8, 2025

[RFC][PoC] Zephyr PreProcessor (ZPP) framework #82990

Are you sure you want to change the base?

[RFC][PoC] Zephyr PreProcessor (ZPP) framework #82990

Conversation

pdgendt commented Dec 14, 2024 • edited Loading

Introduction

Problem description

Proposed change

Detailed RFC

Proposed change (Detailed)

Dependencies

Concerns and Unresolved Questions

Alternatives

henrikbrixandersen commented Dec 16, 2024

pdgendt commented Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

pdgendt commented Dec 16, 2024

gmarull Dec 17, 2024

Choose a reason for hiding this comment

pdgendt Dec 17, 2024

Choose a reason for hiding this comment

gmarull Dec 17, 2024

Choose a reason for hiding this comment

pdgendt Dec 17, 2024

Choose a reason for hiding this comment

gmarull Dec 17, 2024

Choose a reason for hiding this comment

pdgendt Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

bjarki-andreasen commented Dec 17, 2024 • edited Loading

pdgendt commented Dec 23, 2024

RobinKastberg commented Jan 8, 2025

RobinKastberg commented Jan 8, 2025

pdgendt commented Jan 8, 2025

pdgendt commented Dec 14, 2024 •

edited

Loading

pdgendt commented Dec 16, 2024 •

edited

Loading

pdgendt Dec 17, 2024 •

edited

Loading

bjarki-andreasen commented Dec 17, 2024 •

edited

Loading