Compressed sysimage using zstd (manual 1.11 backport) #59340
Conversation
That looks unrelated (was this the magic github issue number autocompletion?) |
|
The PR #59227 should be linked instead, I think. |
topolarity
changed the base branch from
backports-release-1.11
to
release-1.11
topolarity
force-pushed
the
compressed-sysimg-1.11
branch
from
1fe2358 to
82e2dc4
Compare
Inspired by a bug I hit on MinGW recently with weak symbol resolution. I'm not sure why we started using these in 70000ac, since I believe we should be able to lookup these symbols just fine in our `jl_exe_handle` as long as it was linked / compiled with `-rdynamic` Migrating away from weak symbols seems a good idea, given their uneven implementation across platforms. (cherry picked from commit 4a6ada6)
Introduce `jl_image_buf_t` to represent the in-memory details of an unparsed system image and clean-up several global variables and split loading logic in staticdata.c so that we get (almost) everything we need from the sysimage handle at once. This allows sysimage loading to be separated into three phases: 1. Lookup the raw sysimage buffer as a `jl_image_buf_t` 2. For .so sysimages, parse the sysimage and initialize JIT targets, etc. 3. Finally load the sysimage into a `jl_image_t` Care was taken to preserve the existing behavior of calling `jl_set_sysimg_so` to configure the sysimage before initializing Julia, although this is likely to be next on the chopping block in a follow-up. Dependent on JuliaLang#57523.
Co-authored-by: Gabriel Baraldi <[email protected]> madvise with MADV_WILLNEED before accessing sysimage pages MADV_WILLNEED sysimage pages before checksumming or decompressing Even without compression, this gives about an 8% improvement in load times. Fix using jl_page_size before it is initialized Add zstd to ANALYSIS_DEPS now that it is used in staticdata.c Fix jl_page_size being uninitialized in jl_prefetch_system_image Add --compress-sysimage flag; use zstd-15 when enabled
topolarity
force-pushed
the
compressed-sysimg-1.11
branch
from
82e2dc4 to
5c227a2
Compare
|
Looks like the Took the liberty of re-targeting this to |
xal-0
changed the title
This is what I get for trusting the GitHub textbox instead of pasting in text |
|
Adding a stdlib in a patch version seems highly unusual. It would be good to add the motivation for backporting this. |
|
That is true, though the presence of Zstd_jll here is an implementation detail. I'm not familiar enough with the build system to know if it would be easy to link libjulia-internal against libzstd without bringing in the JLL. |
|
I've backported #59498 to avoid any breakage. |
|
I agree that this sounds quite large for a backport. @KristofferC Do you have any thoughts here, on whether this is appropriate to backport to 1.11.x? |
|
I don't think we intend to backport this 1.11 - this PR (marked both draft and |
…59773) Handle a failed allocation in `jl_image_unpack_zstd` better: - On Windows, if `GetLargePageMinimum()` reports that large pages are unsupported, don't try to allocate with `MEM_LARGE_PAGES`. - If they are supported but the large page allocation fails, try again without the option. - On all platforms, provide a better error message when the allocation fails. Ported from #59340.
…59773) Handle a failed allocation in `jl_image_unpack_zstd` better: - On Windows, if `GetLargePageMinimum()` reports that large pages are unsupported, don't try to allocate with `MEM_LARGE_PAGES`. - If they are supported but the large page allocation fails, try again without the option. - On all platforms, provide a better error message when the allocation fails. Ported from #59340.
…59773) Handle a failed allocation in `jl_image_unpack_zstd` better: - On Windows, if `GetLargePageMinimum()` reports that large pages are unsupported, don't try to allocate with `MEM_LARGE_PAGES`. - If they are supported but the large page allocation fails, try again without the option. - On all platforms, provide a better error message when the allocation fails. Ported from #59340.
8 participants
This is #59227 manually backported to 1.11, along with the dependencies #57523,
#57542, and a modified version of #58344 that adds the zstd jll but does not
make any of the LLVM changes.