GH-140479: Update JIT builds to use LLVM 21 #140973

savannahostrowski · 2025-11-04T04:56:24Z

Well, here's a fun treat...looks like the logic to fetch the LLVM binary as a release artifact from https://github.com/python/cpython-bin-deps/releases/tag/llvm-21.1.4.0 never actually worked 😓. CI only passed for Windows on #140329 because LLVM 20 is included in the base image we use for Windows builds (example: https://github.com/actions/partner-runner-images/blob/main/images/arm-windows-11-image.md#language-and-runtime).

As such, we should merge #141002 before this one (Windows CI will fail otherwise!).

Issue: Update JIT builds to use LLVM 21 #140479

…into llvm-21

Tools/jit/README.md

diegorusso · 2025-11-04T10:07:42Z

Well, here's a fun treat...looks like the logic to fetch the LLVM binary as a release artifact from https://github.com/python/cpython-bin-deps/releases/tag/llvm-21.1.4.0 never actually worked 😓. CI only passed for Windows on #140329 because LLVM 20 is included in the base image we use for Windows builds (see https://github.com/actions/partner-runner-images/blob/main/images/arm-windows-11-image.md#language-and-runtime).

This PR patches up the logic for extracting the tarball in get_externals.py and then...just bumps the version to 21.

Issue: Update JIT builds to use LLVM 21 #140479

What about the Windows x86_64 image? Does it have llvm 20 as well?

Also, in this PR we do 2 things: fix some logic and bump to 21. Wouldn't it be better to do 2 PRs? In case we need to revert to llvm 20 (for any reason) we will revert to a working fetching for version 20.

savannahostrowski · 2025-11-04T16:13:39Z

@diegorusso Same deal with x86_64 - see https://github.com/actions/runner-images/blob/win22/20251021.76/images/windows/Windows2022-Readme.md#language-and-runtime

Sure - see #141002

diegorusso

Provided #141002 gets merged before, this can go in as well. Thanks for splitting them up.

Fidget-Spinner

Looks simple enough, let's do this!

pablogsal · 2025-11-09T17:36:50Z

I have tried to rebase https://github.com/pablogsal/cpython-unwind to check the unwinding. Looks like the function to test now needs to be more complicated beause the JIT refuses to call functions if there are branches now and requires tons of iterations to start jitting. I would love if someone can take a look at run.py. With the code we can validate all unwinding methods

Fidget-Spinner · 2025-11-09T20:55:59Z

The main branch's JIT doesn't seem to want to compile the example due to trace stack underflow. So I reduced the side exit count on my tracing JIT example and tried. Seems like LLVM 21 doesn't fix the problem :(

PYTHON_JIT=0
-----------
Testing various unwinding methods on the same deep call stack...

[GNU backtrace]
Total frames: 27
Showing first 10 frames:
  # 0  /home/ken/Documents/GitHub/cpython-unwind/stackunwind.cpython-315-x86_64-linux-gnu.so(+0x280a) [0x78674f7a580a]
  # 1  ../cpython/python(PyObject_Vectorcall+0x33) [0x58c0e08d2303]
  # 2  ../cpython/python(_PyEval_EvalFrameDefault+0xf7a1) [0x58c0e086b141]
  # 3  ../cpython/python(+0x2860aa) [0x58c0e0a430aa]
  # 4  ../cpython/python(PyObject_Vectorcall+0x33) [0x58c0e08d2303]
  # 5  ../cpython/python(_PyEval_EvalFrameDefault+0x3fb3) [0x58c0e085f953]
  # 6  ../cpython/python(+0x2860aa) [0x58c0e0a430aa]
  # 7  ../cpython/python(PyObject_Vectorcall+0x33) [0x58c0e08d2303]
  # 8  ../cpython/python(_PyEval_EvalFrameDefault+0x3fb3) [0x58c0e085f953]
  # 9  ../cpython/python(+0x2860aa) [0x58c0e0a430aa]
  ... (17 more frames)

[libunwind]
Total frames: 26
Showing first 10 frames:
  # 0  PyObject_Vectorcall+0x33
  # 1  _PyEval_EvalFrameDefault+0xf7a1
  # 2  _PyEval_Vector+0x21a
  # 3  PyObject_Vectorcall+0x33
  # 4  _PyEval_EvalFrameDefault+0x3fb3
  # 5  _PyEval_Vector+0x21a
  # 6  PyObject_Vectorcall+0x33
  # 7  _PyEval_EvalFrameDefault+0x3fb3
  # 8  _PyEval_Vector+0x21a
  # 9  PyObject_Vectorcall+0x33
  ... (16 more frames)

[libdw (DWARF)]
Total frames: 26
Showing first 10 frames:
  # 0  PyObject_Vectorcall
  # 1  _PyEval_EvalFrameDefault
  # 2  _PyEval_Vector
  # 3  PyObject_Vectorcall
  # 4  _PyEval_EvalFrameDefault
  # 5  _PyEval_Vector
  # 6  PyObject_Vectorcall
  # 7  _PyEval_EvalFrameDefault
  # 8  _PyEval_Vector
  # 9  PyObject_Vectorcall
  ... (16 more frames)

[Manual frame pointers]
Total frames: 4
Showing first 4 frames:
  # 0  0x58c0e08d2303
  # 1  0x2
  # 2  0x58c0e0dbb7e0
  # 3  0x2


PYTHON_JIT=1
------------
[GNU backtrace]
Total frames: 6
Showing first 6 frames:
  # 0  /home/ken/Documents/GitHub/cpython-unwind/stackunwind.cpython-315-x86_64-linux-gnu.so(+0x280a) [0x7a0a357d080a]
  # 1  ../cpython/python(PyObject_Vectorcall+0x33) [0x57385e3d3303]
  # 2  ../cpython/python(_PyEval_EvalFrameDefault+0xf7a1) [0x57385e36c141]
  # 3  ../cpython/python(+0x2860aa) [0x57385e5440aa]
  # 4  ../cpython/python(PyObject_Vectorcall+0x33) [0x57385e3d3303]
  # 5  [0x7a0a3398d4fb]

[libunwind]
Total frames: 5
Showing first 5 frames:
  # 0  PyObject_Vectorcall+0x33
  # 1  _PyEval_EvalFrameDefault+0xf7a1
  # 2  _PyEval_Vector+0x21a
  # 3  PyObject_Vectorcall+0x33
  # 4  <unknown>

[libdw (DWARF)]
Total frames: 5
Showing first 5 frames:
  # 0  PyObject_Vectorcall
  # 1  _PyEval_EvalFrameDefault
  # 2  _PyEval_Vector
  # 3  PyObject_Vectorcall
  # 4  <unknown>

[Manual frame pointers]
Total frames: 4
Showing first 4 frames:
  # 0  0x57385e3d3303
  # 1  0x(nil)
  # 2  0x57385e8bc7e0
  # 3  0x(nil)

Notice the missing frames!

However, I have made a very interesting discovery --- compiling the interpreter with a recent enough clang version (tested with clang-20/21) to the JIT restores the frames!!!

[GNU backtrace]
Total frames: 29
Showing first 10 frames:
  # 0  /home/ken/Documents/GitHub/cpython-unwind/stackunwind.cpython-315-x86_64-linux-gnu.so(+0x280a) [0x797cf3f7c80a]
  # 1  ../cpython/python(PyObject_Vectorcall+0x43) [0x64be6334e693]
  # 2  ../cpython/python(_PyEval_EvalFrameDefault+0x5d3e) [0x64be6347f63e]
  # 3  ../cpython/python(+0x217706) [0x64be63479706]
  # 4  ../cpython/python(PyObject_Vectorcall+0x43) [0x64be6334e693]
  # 5  ../cpython/python(_PyEval_EvalFrameDefault+0x3768) [0x64be6347d068]
  # 6  ../cpython/python(+0x217706) [0x64be63479706]
  # 7  ../cpython/python(PyObject_Vectorcall+0x43) [0x64be6334e693]
  # 8  ../cpython/python(_PyEval_EvalFrameDefault+0x3768) [0x64be6347d068]
  # 9  ../cpython/python(+0x217706) [0x64be63479706]
  ... (19 more frames)

[libunwind]
Total frames: 28
Showing first 10 frames:
  # 0  PyObject_Vectorcall+0x43
  # 1  _PyEval_EvalFrameDefault+0x5d3e
  # 2  _PyEval_Vector+0x266
  # 3  PyObject_Vectorcall+0x43
  # 4  _PyEval_EvalFrameDefault+0x3768
  # 5  _PyEval_Vector+0x266
  # 6  PyObject_Vectorcall+0x43
  # 7  _PyEval_EvalFrameDefault+0x3768
  # 8  _PyEval_Vector+0x266
  # 9  PyObject_Vectorcall+0x43
  ... (18 more frames)

[libdw (DWARF)]
Total frames: 28
Showing first 10 frames:
  # 0  PyObject_Vectorcall
  # 1  _PyEval_EvalFrameDefault
  # 2  _PyEval_Vector
  # 3  PyObject_Vectorcall
  # 4  _PyEval_EvalFrameDefault
  # 5  _PyEval_Vector
  # 6  PyObject_Vectorcall
  # 7  _PyEval_EvalFrameDefault
  # 8  _PyEval_Vector
  # 9  PyObject_Vectorcall
  ... (18 more frames)

[Manual frame pointers]
Total frames: 16
Showing first 10 frames:
  # 0  0x64be6334e693
  # 1  0x2
  # 2  0x64be63479706
  # 3  0x2
  # 4  0x64be63479706
  # 5  0x2
  # 6  0x64be63479706
  # 7  0x(nil)
  # 8  0x64be63479706
  # 9  0x(nil)
  ... (6 more frames)

So the solution to the stack unwinding seems to be to mandate compiilng CPython with the same clang as the JIT!

pablogsal · 2025-11-09T20:58:19Z

So the solution to the stack unwinding seems to be to mandate compiilng CPython with the same clang as the JIT!

We need to understand what the hell it's going on here because otherwise the risk it's that this breaks randomly in the future

Fidget-Spinner · 2025-11-09T21:02:47Z

Another data point: old GCC (11) compiled with frame pointers with JIT also allows us to see the frames.

brandtbucher · 2025-11-10T02:36:21Z

Just to be clear, merely upgrading to LLVM 21 isn’t sufficient to get unwinding. We also need to do the reserved frame pointer thing, which isn’t in this branch.

brandtbucher · 2025-11-10T02:36:41Z

That can come after.

savannahostrowski · 2025-11-10T17:37:36Z

Yeah, was chatting with Brandt earlier this weekend about this and will probably go investigate the unwinding piece after this gets merged.

Fidget-Spinner · 2025-11-10T17:55:10Z

I think it would be good to merge this soon. The meta benchmarking runners don't have JIT results this week because they don't have LLVM 20. It would be a little bit of churn to request they upgrade twice (once for LLVM 20, once for LLVM 21), so I was thinking if I could ask them nicely to just upgrade to LLVM 21 once :).

savannahostrowski · 2025-11-12T17:13:05Z

Discussed in our weekly sync, merging 🚀 !

savannahostrowski added 9 commits November 3, 2025 15:57

Initial commit for LLVM 21

7e7ebaf

Update CI

ae4a01d

Another CI fix

da9b795

Add debugging

d079df9

Add fix for exe

322ea9b

Update tarball extraction

27a91ac

Fix for windows

2ca8603

Clean up after debugging

39e1339

Uncomment CI

38105ce

savannahostrowski added the build The build process and cross-build label Nov 4, 2025

savannahostrowski requested a review from brandtbucher as a code owner November 4, 2025 04:56

savannahostrowski added the dependencies Pull requests that update a dependency file label Nov 4, 2025

savannahostrowski requested review from a team, diegorusso, ezio-melotti and hugovk as code owners November 4, 2025 04:56

savannahostrowski added the topic-JIT label Nov 4, 2025

savannahostrowski requested a review from AA-Turner as a code owner November 4, 2025 04:56

savannahostrowski added the infra CI, GitHub Actions, buildbots, Dependabot, etc. label Nov 4, 2025

bedevere-app bot added the awaiting core review label Nov 4, 2025

bedevere-app bot mentioned this pull request Nov 4, 2025

Update JIT builds to use LLVM 21 #140479

Closed

📜🤖 Added by blurb_it.

e85c8f8

savannahostrowski mentioned this pull request Nov 4, 2025

Bump to Fedora 43 for JIT builds with LLVM 21 python/cpython-devcontainers#62

Merged

savannahostrowski added 2 commits November 3, 2025 21:01

Fix README

3ca9733

Merge branch 'llvm-21' of https://github.com/savannahostrowski/cpython …

4508fc5

…into llvm-21

savannahostrowski commented Nov 4, 2025

View reviewed changes

Tools/jit/README.md Show resolved Hide resolved

savannahostrowski mentioned this pull request Nov 4, 2025

GH-136895: Fixes for pulling LLVM as a release artifact #141002

Merged

Remove fixes for Windows release artifacts

ef52c5a

diegorusso approved these changes Nov 4, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Nov 4, 2025

Merge branch 'main' into llvm-21

9778bdd

Fidget-Spinner approved these changes Nov 7, 2025

View reviewed changes

Merge branch 'main' into llvm-21

39438c0

savannahostrowski enabled auto-merge (squash) November 12, 2025 17:13

savannahostrowski merged commit d162c42 into python:main Nov 12, 2025
109 of 111 checks passed

bedevere-app bot removed the awaiting merge label Nov 12, 2025

Uh oh!

GH-140479: Update JIT builds to use LLVM 21 #140973

GH-140479: Update JIT builds to use LLVM 21 #140973

Conversation

savannahostrowski commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

diegorusso commented Nov 4, 2025

Uh oh!

savannahostrowski commented Nov 4, 2025

Uh oh!

diegorusso left a comment

Choose a reason for hiding this comment

Uh oh!

Fidget-Spinner left a comment

Choose a reason for hiding this comment

Uh oh!

pablogsal commented Nov 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fidget-Spinner commented Nov 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal commented Nov 9, 2025

Uh oh!

Fidget-Spinner commented Nov 9, 2025

Uh oh!

brandtbucher commented Nov 10, 2025

Uh oh!

brandtbucher commented Nov 10, 2025

Uh oh!

savannahostrowski commented Nov 10, 2025

Uh oh!

Fidget-Spinner commented Nov 10, 2025

Uh oh!

savannahostrowski commented Nov 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

savannahostrowski commented Nov 4, 2025 •

edited

Loading

pablogsal commented Nov 9, 2025 •

edited

Loading

Fidget-Spinner commented Nov 9, 2025 •

edited

Loading