Build setup of 1.11.0, use of npm

[stblassitude]

• Gitea version (or commit ref):1.11.0
• Git version:
• Operating system: FreeBSD
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

Description

I'm the maintainer for the FreeBSD port of Gitea, allowing FreeBSD users to simply pkg install gitea.

It seems that with 1.11.0, the CSS and JS files are not included pre-built in the repo anymore, but are built during the Gitea build process, requiring the use of npm install, including network access to download almost a thousand node modules.

The FreeBSD package build system does not allow network access during the build phase; all required files need to be downloaded first using a build-system specific download mechanism from pre-recorded URLs. Among other aspects, this allows the build system to verify cryptographic checksums on files downloaded from potentially unreliable mirrors for source packages.

The FreeBSD ports/package system currently contains no provisions to handle npm install as part of a package build.

This means that with the current state, no FreeBSD package of Gitea 1.11.0 can be produced.

Would you consider adding ./node_modules to the source archive for releases, or provide a single file archive for download of it (matching the release), so that the necessary node modules can be made available to package builds without the necessity to run npm install? I don't think it's necessary to add the directory to the Git repo, only make it part of the release tar.gz. I see this as similar to the vendoring of Go modules.

I have a suspicion that other OS/distributions might have similar restrictions on files that are used during package builds.

[silverwind]

I could see the option of having a separate download for a gitea-$release-node_modules.gzip for this use case. It's important that the extracted node_modules folder's timestamp is newer than package-lock.json, otherwise the build process will try to contact the npm registry, so that file needs to be generated after the main build process.

I do have to wonder if there are more general solutions to this problem. We do use package-lock.json and the dependency packages on npm can be considered immutable, so build should always be exactly reproducible. I know some distributions do package individual npm modules, but I've never been a fan of that.

[guillep2k]

However, not all go modules are vendored. As much as we can include the generated js and css files, I think network access is still required to resolve go dependencies. How was that resolved in previous versions?

[stblassitude]

However, not all go modules are vendored. As much as we can include the generated js and css files, I think network access is still required to resolve go dependencies. How was that resolved in previous versions?

As far as I'm aware, all Go module dependencies are vendored, at least up to 1.10.3.

[lafriks]

True, go modules are included. We could indeed publish node_modules to download folder beside gitea binaries. That could also speedup our own builds with this plugin and s3 configuration: http://plugins.drone.io/meltwater/drone-cache/

[silverwind]

Go modules are vendored currently, but I think the plan is to eventually un-vendor them (Go 1.14 or later), so they too would then need a solution for a "offline" build.

[lafriks]

We can cache them to Gitea dl s3 also then

[lunny]

We can add a sub directory /cache which includes /cache/v1.11.0/go_modules/, /cache/v1.11.0/node_modules/ on dl.gitea.io

[sapk]

I started earlier #9733 it need to be changed to only update cache on master or version branch to limit cache poisonning.

[ecsgh]

Same problem in openSUSE.
I think all build systems does not allow download external files.
All go modules are vendored and includes in source.
But all nodejs packages are not included.
I think nodejs modules must also vendored like go modules.

And please, don't stop to include this modules.
There are needed for building in build systems.
Without that, gitea could no more include in repositories and distributions without extremly more work and "tricks".

[silverwind]

Vendoring node_modules would currently add 36231 files at 276 MB (42M gzipped) to the repo.

I still wonder why distributions need their build systems to be offline.

[sapk]

Maybe a .tar.gz (for each tag) of source code with vendor inside (go and node) is good for you ? This way the build is reproducible from those archive.

[silverwind]

I just checked what Kibana (who is also a heavy node modules user): They don't have node_modules in source control but their release source tarballs do include node_modules.

So I think we just need to update the release process to create a source tarball with modules included. One has to keep in mind that they can be platform-specific when native node.js modules are present, but luckily we don't really have any of those (besides the optional fsevents on macOS), so our node_modules should be generally portable.

[stblassitude]

Including the modules in the release source tarballs would be perfect!

[rnhmjoj]

I still wonder why distributions need their build systems to be offline.

Reproducibility.