west.yml: lazy module loading

[cfriedt]

Introduction

This topic should really be of interest for anyone who uses west to pull in Zephyr components, which is likely a large majority at this time.

Generally though, the topic should be of interest to end users, TSC members, maintainers, product designers, module authors, etc.

Problem description

Modules are the primary vehicle used to scale Zephyr.

At the beginning, it was fairly inconsequential that there were 5-10 modules pulling in maybe ~100 MB of additional repositories, HALs, libraries, etc. However, as Zephyr and the Zephyr Community have grown, it's become a more frequent concern that the default behaviour (west update) pulls in several GB of repository data. Moreover, for the a common goal like running hello_world on a particular board, the vast majority of modules, and the GB of git repository data downloaded, are largely unused.

There was previously an attempt to push the problem onto users by grouping west repositories into categories (core, hal, etc). However, that does not seem to have worked as well we had hoped.

Proposed change

In an ideal world, everything would be optional. Our default should be to minimize the required packages to install and the required modules to download.

We propose something along the lines of "lazy module loading". I.e. automatically updating only the the git repositories necessary for a particular app using a particular configuration running on a particular board.

Automatically downloading dependencies during a build phase are generally not a great idea, so this should be an optional feature that is disabled by default.

Some potentially "nice" features to have would be

• maybe prompt users to download repos (if in an interactive shell)?
• maybe something like a "--fetch-only" option that does the fetching in a separate step for a given app / board / config
• maybe something like a "--dry-run" print out the list of repos that would need to be downloaded, so that people can manually do that in advance for their particular CI systems
• maybe something like a "deps --list" option

Additional notes

• It's not just west modules, but the tools (e.g. ubuntu / python packages) needed to deal with those modules
  • augment module yaml schema with necessary packages
    • ubuntu: (array of deb package-names..)
    • macos: (array of package-names..)
    • windows: (array of package-names..)
    • pip: (array of package-names..)
    • redhat: (array of package-names..)
  • each of the above would of course be optional, but every module would be required to list its own dependencies outside of the most basic requirements for building Zephyr.

• West update tooling dependency management #55056

Detailed RFC

TODO

Proposed change (Detailed)

TODO

Dependencies

TODO

Concerns and Unresolved Questions

TODO

Alternatives

Deja-vu - we have had discussions and solutions about this several times.

[cfriedt]

Some PRs with associated conversations about this topic

• thrift: module integration, samples, and tests #54013
• Introduce Pigweed Module #53760

[galak]

I think one question is what kinda of means do we convey how a given module is needed. For example, is that via some Kconfig symbols?

[cfriedt]

Also, just throwing this out there, but we could consider this as a problem for a GSoC contributor 🤷

[mgielda]

Thanks @cfriedt for opening up the issue. I think it's probably more important to us than a GSoC proposal, which after all 1) will not necessarily be picked and 2) can fail (not that we can't have a GSoC about it, but then probably we should wait until the GSoC completes with undertaking other attempts; I assume the ideal GSoC project should not be on a critical path, just to avoid unnecessary pressure on the student).

I am for an apt-like behaviour w/r to asking the user - i.e. by default it will stop on recomending packages to download and wait for approval, you have to explicitly override that behavior but then everyone assumes you know what you are doing. A --download-all flag would probably make sense too.

For determining modules, Kconfig makes sense to me too.

[cfriedt]

Yea, I just won't have time to do this myself. So cannot be the assignee unfortunately. A GSoC contributor would be great in terms of someone to offload work to. Even if a GSoC fails, the work could still be used, etc. But yea, it might not necessarily be picked.

Getting this right will not be an overnight problem though, by any means. Given that we have 55 modules right now, I would expect that part of the task would be to itemize the packages that each module depends on (pip, .deb, etc). That alone could take a considerable amount of time.

[mbolivar-nordic]

@MaureenHelm MaureenHelm assigned mbolivar-nordic 5 days ago

I welcome someone developing a package manager based on west, but I have disclaimed this assignment already elsewhere.

@cfriedt feel free to reopen and assign yourself or someone else if you're aware of a volunteer.