Skip to content

staticdata: Refactor sysimage loading #57542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Feb 28, 2025
Merged

staticdata: Refactor sysimage loading #57542

merged 2 commits into from
Feb 28, 2025

Conversation

@topolarity
Copy link
Member

@topolarity topolarity commented Feb 26, 2025 •
edited
Loading

Introduce jl_image_buf_t to represent the in-memory details of an unparsed system image and clean-up several global variables and split loading logic in staticdata.c so that we get (almost) everything we need from the sysimage handle at once.

This allows sysimage loading to be separated into three phases:

  1. Lookup the raw sysimage buffer as a jl_image_buf_t
  2. For .so sysimages, parse the sysimage and initialize JIT targets, etc.
  3. Finally load the sysimage into a jl_image_t

Care was taken to preserve the existing behavior of calling jl_set_sysimg_so to configure the sysimage before initializing Julia, although this is likely to be next on the chopping block in a follow-up.

Dependent on #57523.

@topolarity
staticdata: Refactor sysimage loading
cca51ef 
Introduce `jl_image_buf_t` to represent the in-memory details of an unparsed
system image and clean-up several global variables and split loading logic
in staticdata.c so that we (almost) get everything we need from the sysimage
handle at once.

This allows sysimage loading to be separated into three phases:
  1. Lookup the raw sysimage buffer as a `jl_image_buf_t`
  2. For .so sysimages, parse the sysimage and initialize JIT targets, etc.
  3. Finally load the sysimage into a `jl_image_t`

Care was taken to preserve the existing behavior of calling `jl_set_sysimg_so`
to configure the sysimage before initializing Julia, although this is likely
to be next on the chopping block in a follow-up.
vtjnash
vtjnash approved these changes Feb 27, 2025
View reviewed changes
Copy link
Member

@vtjnash vtjnash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this feels less magic and global coupling of hidden logic. Much clearer now

@topolarity topolarity force-pushed the ct/init-refactor branch 2 times, most recently from 9079057 to d4dbb95 Compare February 27, 2025 19:35
@topolarity
Pass around jl_options.cpu_target explicitly
e0bde91 
This (mostly) removes the only usage of `jl_options` internally in
`processor_*.cpp`
@Keno Keno merged commit ad61241 into JuliaLang:master Feb 28, 2025
7 checks passed
@KristofferC KristofferC added the backport 1.12 Change should be backported to release-1.12 label May 5, 2025
@KristofferC KristofferC mentioned this pull request May 5, 2025
Merged
53 tasks
gbaraldi added a commit that referenced this pull request May 5, 2025
@gbaraldi
staticdata: Refactor sysimage loading (#57542)
eb09ae1 
Introduce `jl_image_buf_t` to represent the in-memory details of an
unparsed system image and clean-up several global variables and split
loading logic in staticdata.c so that we get (almost) everything we need
from the sysimage handle at once.

This allows sysimage loading to be separated into three phases:
  1. Lookup the raw sysimage buffer as a `jl_image_buf_t`
2. For .so sysimages, parse the sysimage and initialize JIT targets,
etc.
  3. Finally load the sysimage into a `jl_image_t`

Care was taken to preserve the existing behavior of calling
`jl_set_sysimg_so` to configure the sysimage before initializing Julia,
although this is likely to be next on the chopping block in a follow-up.

Dependent on #57523.
@gbaraldi gbaraldi mentioned this pull request May 5, 2025
Merged
@KristofferC KristofferC removed the backport 1.12 Change should be backported to release-1.12 label May 9, 2025
xal-0 pushed a commit to xal-0/julia that referenced this pull request Aug 18, 2025
@topolarity @xal-0
staticdata: Refactor sysimage loading (JuliaLang#57542)
4b731bb 
Introduce `jl_image_buf_t` to represent the in-memory details of an
unparsed system image and clean-up several global variables and split
loading logic in staticdata.c so that we get (almost) everything we need
from the sysimage handle at once.

This allows sysimage loading to be separated into three phases:
  1. Lookup the raw sysimage buffer as a `jl_image_buf_t`
2. For .so sysimages, parse the sysimage and initialize JIT targets,
etc.
  3. Finally load the sysimage into a `jl_image_t`

Care was taken to preserve the existing behavior of calling
`jl_set_sysimg_so` to configure the sysimage before initializing Julia,
although this is likely to be next on the chopping block in a follow-up.

Dependent on JuliaLang#57523.
xal-0 added a commit that referenced this pull request Aug 20, 2025
@xal-0
System image compression with zstd (1.11 backport)
b9bcdb3 
commit d062060
Author: Sam Schweigel <[email protected]>
Date:   Wed Aug 6 16:35:34 2025 -0700

    System image compression with zstd

    Co-authored-by: Gabriel Baraldi <[email protected]>

    madvise with MADV_WILLNEED before accessing sysimage pages

    MADV_WILLNEED sysimage pages before checksumming or decompressing

    Even without compression, this gives about an 8% improvement in load times.

    Fix using jl_page_size before it is initialized

    Add zstd to ANALYSIS_DEPS now that it is used in staticdata.c

    Fix jl_page_size being uninitialized in jl_prefetch_system_image

    Add --compress-sysimage flag; use zstd-15 when enabled

commit 56a6065
Author: Sam Schweigel <[email protected]>
Date:   Mon Aug 18 13:27:13 2025 -0700

    Janky zstd backport

commit 4b731bb
Author: Cody Tapscott <[email protected]>
Date:   Thu Feb 27 22:20:02 2025 -0500

    staticdata: Refactor sysimage loading (#57542)

    Introduce `jl_image_buf_t` to represent the in-memory details of an
    unparsed system image and clean-up several global variables and split
    loading logic in staticdata.c so that we get (almost) everything we need
    from the sysimage handle at once.

    This allows sysimage loading to be separated into three phases:
      1. Lookup the raw sysimage buffer as a `jl_image_buf_t`
    2. For .so sysimages, parse the sysimage and initialize JIT targets,
    etc.
      3. Finally load the sysimage into a `jl_image_t`

    Care was taken to preserve the existing behavior of calling
    `jl_set_sysimg_so` to configure the sysimage before initializing Julia,
    although this is likely to be next on the chopping block in a follow-up.

    Dependent on #57523.
xal-0 added a commit to xal-0/julia that referenced this pull request Aug 20, 2025
@xal-0
System image compression with zstd (1.11 backport)
27d4397 
commit d062060
Author: Sam Schweigel <[email protected]>
Date:   Wed Aug 6 16:35:34 2025 -0700

    System image compression with zstd

    Co-authored-by: Gabriel Baraldi <[email protected]>

    madvise with MADV_WILLNEED before accessing sysimage pages

    MADV_WILLNEED sysimage pages before checksumming or decompressing

    Even without compression, this gives about an 8% improvement in load times.

    Fix using jl_page_size before it is initialized

    Add zstd to ANALYSIS_DEPS now that it is used in staticdata.c

    Fix jl_page_size being uninitialized in jl_prefetch_system_image

    Add --compress-sysimage flag; use zstd-15 when enabled

commit 56a6065
Author: Sam Schweigel <[email protected]>
Date:   Mon Aug 18 13:27:13 2025 -0700

    Janky zstd backport

commit 4b731bb
Author: Cody Tapscott <[email protected]>
Date:   Thu Feb 27 22:20:02 2025 -0500

    staticdata: Refactor sysimage loading (JuliaLang#57542)

    Introduce `jl_image_buf_t` to represent the in-memory details of an
    unparsed system image and clean-up several global variables and split
    loading logic in staticdata.c so that we get (almost) everything we need
    from the sysimage handle at once.

    This allows sysimage loading to be separated into three phases:
      1. Lookup the raw sysimage buffer as a `jl_image_buf_t`
    2. For .so sysimages, parse the sysimage and initialize JIT targets,
    etc.
      3. Finally load the sysimage into a `jl_image_t`

    Care was taken to preserve the existing behavior of calling
    `jl_set_sysimg_so` to configure the sysimage before initializing Julia,
    although this is likely to be next on the chopping block in a follow-up.

    Dependent on JuliaLang#57523.
xal-0 pushed a commit to xal-0/julia that referenced this pull request Aug 20, 2025
@topolarity @xal-0
staticdata: Refactor sysimage loading (JuliaLang#57542)
8150fe2 
Introduce `jl_image_buf_t` to represent the in-memory details of an
unparsed system image and clean-up several global variables and split
loading logic in staticdata.c so that we get (almost) everything we need
from the sysimage handle at once.

This allows sysimage loading to be separated into three phases:
  1. Lookup the raw sysimage buffer as a `jl_image_buf_t`
2. For .so sysimages, parse the sysimage and initialize JIT targets,
etc.
  3. Finally load the sysimage into a `jl_image_t`

Care was taken to preserve the existing behavior of calling
`jl_set_sysimg_so` to configure the sysimage before initializing Julia,
although this is likely to be next on the chopping block in a follow-up.

Dependent on JuliaLang#57523.
@xal-0 xal-0 mentioned this pull request Aug 20, 2025
Draft
topolarity added a commit to topolarity/julia that referenced this pull request Aug 20, 2025
@topolarity
staticdata: Refactor sysimage loading (JuliaLang#57542)
89de154 
Introduce `jl_image_buf_t` to represent the in-memory details of an
unparsed system image and clean-up several global variables and split
loading logic in staticdata.c so that we get (almost) everything we need
from the sysimage handle at once.

This allows sysimage loading to be separated into three phases:
  1. Lookup the raw sysimage buffer as a `jl_image_buf_t`
2. For .so sysimages, parse the sysimage and initialize JIT targets,
etc.
  3. Finally load the sysimage into a `jl_image_t`

Care was taken to preserve the existing behavior of calling
`jl_set_sysimg_so` to configure the sysimage before initializing Julia,
although this is likely to be next on the chopping block in a follow-up.

Dependent on JuliaLang#57523.
topolarity added a commit to xal-0/julia that referenced this pull request Aug 20, 2025
@topolarity
staticdata: Refactor sysimage loading (JuliaLang#57542)
b4c714a 
Introduce `jl_image_buf_t` to represent the in-memory details of an
unparsed system image and clean-up several global variables and split
loading logic in staticdata.c so that we get (almost) everything we need
from the sysimage handle at once.

This allows sysimage loading to be separated into three phases:
  1. Lookup the raw sysimage buffer as a `jl_image_buf_t`
2. For .so sysimages, parse the sysimage and initialize JIT targets,
etc.
  3. Finally load the sysimage into a `jl_image_t`

Care was taken to preserve the existing behavior of calling
`jl_set_sysimg_so` to configure the sysimage before initializing Julia,
although this is likely to be next on the chopping block in a follow-up.

Dependent on JuliaLang#57523.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vtjnash vtjnash vtjnash approved these changes

@gbaraldi gbaraldi Awaiting requested review from gbaraldi

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@topolarity @vtjnash @KristofferC @Keno