Proprietary toolchain support sustainability

[fabiobaltieri]

Introduction

Problem description

Hi, I recently bumped into few compiler quirks:

• the first priority check implementation broke ARCMWDT due to differences in section naming
• the second implementation broke ARMClang due to missing symbols
• the troubleshooting of the second implementation was slowed down because ARMClang support was already broken
• a recent build time optimization added a warning in clang builds for pretty much anything but native_posix that turned out to be an llvm bug
• we are now blocking a Kconfig option removal for unclear compiler limitations (may not be the case, but it adds to the point)

None of the above were detectable in CI. On paper the project supports a handful of different compilers, but in practice the only ones tested in CI are

• Zephyr SDK at the current stable version
• Clang/LLVM at the version in the CI image, but really only builds for native_posix

Any breakage that does not hit any of these is going to go undetected and has to get caught manually by a reviewer or post-submit, potentially post-release, so it's regression and potential backport.

More importantly, many of those toolchains are proprietary, most developer have no access to them, so one has to rely on someone else to test their code, but at the same time there's not toolchain maintainers or a known list of point of contacts for validating changes against a specific toolchain.

Proposed change

• Supported proprietary toolchains should have an area in the MAINTAINER file listing users who can test potentially breaking changes. Ideally there should be a way for Zephyr contributors to access proprietary toolchains for troubleshooting, or the support team should provide help in running the changes on the toolchain and providing build artifacts, or contributing fixes and workarounds directly.
• Supported open source toolchains should have some level of testing in CI (Clang/LLVM beyond native_posix is what I'm thinking)

[fabiobaltieri]

cc @cfriedt @evgeniy-paltsev @galak @gmarull @karben @keith-zephyr @kokas-a @nashif @tejlmand

[cfriedt]

At one point, we spoke about reporting test results from trusted test runners outside of Zephyr CI such as SoC vendors like ST or NXP.

The toolchain spec should be an integral part of that. Possibly an enumeration. Custom LLVM and custom GCC should also be a thing. We have for example CROSS_COMPILE, but it would be great to capture more metadata than that for reporting purposes.

Like external modules, external toolchains would need to run a subset of tests (often proprietary toolchains only support a limited number of platforms).

Maybe we could reuse some of that machinery

[fabiobaltieri]

At one point, we spoke about reporting test results from trusted test runners outside of Zephyr CI such as SoC vendors like ST or NXP.

Would it make sense to investigate the possibility of having some custom runners with those toolchains available running a limited set of core tests? Something on the order of magnitude of the current Clang/LLVM one.

[tejlmand]

Would it make sense to investigate the possibility of having some custom runners with those toolchains available running a limited set of core tests? Something on the order of magnitude of the current Clang/LLVM one.

it makes a lot of sense, and has been discussed in Toolchain WG, but guess it mostly has to do with priorities.
So if anyone have the skills and bandwidth to carry out such work, then it will be highly appreciated.

[tejlmand]

please find the following proposal as a starting point: #63759

Hopefully this will provide more visible information on whom to contact in case of issues related to those toolchains.
Not the final solution, but a step in a better direction.

[abrodkin]

I do support that discussion here as we (Synopsys ARC maintainers) and our customers are suffering from exactly that problem: even though we do quite extensive in-house testing with our proprietary tools, simulators and boards, we only do it post-merge upstream. In that sense we'd prefer to enable 3rd-party developers to see problems which are introduced by their changes before merging these changes in the upstream main tree.

But then we start seeing logistical problems:

1. From one point of view it's hard to make proprietary tools and even more so HW boards freely available to the wide audience. There're usually legal and availability/scalability problems.
2. From another point of view all the above difficulties could be solved in one or another way, but that requires some work to be done. For example, come-up with extensible Zephyr CI pipeline architecture so that external (vendor managed) runners could be involved. And on top of the initial implementation we'll get problems with stability (i.e. we'll need troubleshoot things in those external runners) and, again, scalability (even if we may allocate 10 boards, it will be nowhere close to required bandwidth).

That said I fully agree with @tejlmand here: if there's somebody who's willing to bite that bullet and work on that, we'll provide all the support with reviewing, testing etc.