Add support for Python 3.13

[edmorley]

Add support for Python 3.13, and release 3.13.0.

The default Python version remains unchanged (at 3.12.x) for now.

Notably for Python 3.13 we now:

• No longer install setuptools and wheel - matching what the wider ecosystem has already done for Python 3.12+. (See the Python CNB's removal PR for more details: Stop installing setuptools and wheel buildpacks-python#243)
• No longer install the SQLite headers and CLI, as the first step towards dropping that rarely used feature.
• Have to disable some of the tests that run during the binary build's PGO profiling phase when on Heroku-22 (see comments below for more details).

In addition, for all Python versions we now also remove the idle3 and pydoc3 scripts, since they do not work with relocated Python and so have been broken for some time. Their functionality continues to be available by invoking them via their modules instead (e.g. python -m pydoc).

Release announcement:
https://blog.python.org/2024/10/python-3130-final-released.html
https://www.python.org/downloads/release/python-3130/

Details on what's new in Python 3.13:
https://docs.python.org/3.13/whatsnew/3.13.html

Binary builds:
https://github.com/heroku/heroku-buildpack-python/actions/runs/11280580537

Python 3.13 readiness status of the top 360 packages on PyPI: https://pyreadiness.org/3.13/

GUS-W-14846826.
GUS-W-14846839.
GUS-W-16944574.

[edmorley]

The binary builds of Python 3.13.0 succeeded on Heroku-20 and Heroku-24, however, failed on Heroku-22 (Ubuntu 22.04) due to failures during the Python tests that are run as part of generating the PGO profile:

0:01:48 load avg: 0.93 [43/44] test_xml_etree
test test_xml_etree failed
0:01:49 load avg: 0.93 [44/44] test_xml_etree_c -- test_xml_etree failed (3 failures)
test test_xml_etree_c failed
test_xml_etree_c failed (3 failures)

Sadly that is all the output shown. (Build logs at: https://github.com/heroku/heroku-buildpack-python/actions/runs/11259777638/job/31310276989#step:4:1374)

A retrigger of the build didn't resolve the issue.

The offending tests are these:
https://github.com/python/cpython/blob/v3.13.0/Lib/test/test_xml_etree.py
https://github.com/python/cpython/blob/v3.13.0/Lib/test/test_xml_etree_c.py

[edmorley]

This might be related to:
python/cpython#125067

[edmorley]

Looking back at the builds for older Python versions (eg those for 3.12) shows that the PGO profile test suite reports failures on Heroku-22 there too. The difference is that for Python 3.13 any failures in the tests during the PGO profile generation step are now no longer ignored after:
python/cpython#110295

To see the full test failure output, I modified our build script to pass "PROFILE_TASK=-m test -v test_xml_etree test_xml_etree_c" (to run just the two affected tests, plus with -v for verbose) to the make invocation, per:
https://docs.python.org/3/using/configure.html#envvar-PROFILE_TASK
https://devguide.python.org/testing/run-write-tests/

This resulted in these logs:

======================================================================
FAIL: test_flush_reparse_deferral_disabled (test.test_xml_etree.XMLPullParserTest.test_flush_reparse_deferral_disabled)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/src/Lib/test/test_xml_etree.py", line 1752, in test_flush_reparse_deferral_disabled
    self.assert_event_tags(parser, [('start', 'doc')])
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/src/Lib/test/test_xml_etree.py", line 1483, in assert_event_tags
    self.assertEqual([(action, elem.tag) for action, elem in events],
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                     expected)
                     ^^^^^^^^^
AssertionError: Lists differ: [] != [('start', 'doc')]

Second list contains 1 additional elements.
First extra element 0:
('start', 'doc')

- []
+ [('start', 'doc')]

======================================================================
FAIL: test_simple_xml_chunk_1 (test.test_xml_etree.XMLPullParserTest.test_simple_xml_chunk_1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/src/Lib/test/test_xml_etree.py", line 1508, in test_simple_xml_chunk_1
    self.test_simple_xml(chunk_size=1, flush=True)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/src/Lib/test/test_xml_etree.py", line 1496, in test_simple_xml
    self.assert_event_tags(parser, [('end', 'element')])
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/src/Lib/test/test_xml_etree.py", line 1483, in assert_event_tags
    self.assertEqual([(action, elem.tag) for action, elem in events],
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                     expected)
                     ^^^^^^^^^
AssertionError: Lists differ: [] != [('end', 'element')]

Second list contains 1 additional elements.
First extra element 0:
('end', 'element')

- []
+ [('end', 'element')]

======================================================================
FAIL: test_simple_xml_chunk_5 (test.test_xml_etree.XMLPullParserTest.test_simple_xml_chunk_5)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/src/Lib/test/test_xml_etree.py", line 1511, in test_simple_xml_chunk_5
    self.test_simple_xml(chunk_size=5, flush=True)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/src/Lib/test/test_xml_etree.py", line 1496, in test_simple_xml
    self.assert_event_tags(parser, [('end', 'element')])
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/src/Lib/test/test_xml_etree.py", line 1483, in assert_event_tags
    self.assertEqual([(action, elem.tag) for action, elem in events],
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                     expected)
                     ^^^^^^^^^
AssertionError: Lists differ: [] != [('end', 'element')]

Second list contains 1 additional elements.
First extra element 0:
('end', 'element')

- []
+ [('end', 'element')]

----------------------------------------------------------------------
Ran 198 tests in 0.898s

FAILED (failures=3, skipped=6)
test test_xml_etree failed

(And similar for test_xml_etree_c)

(See: https://github.com/heroku/heroku-buildpack-python/actions/runs/11275986127/job/31358712256#step:4:1547)

[edmorley]

I've added a patch that disables the affected tests for now, since:

1. These tests are only used during PGO profile generation, so are just a way that the build process uses to identify hot paths, so running 3 fewer testcases out of the hundreds won't make a noticeable difference.
2. These tests are failing on Ubuntu 22.04 for older Python too - the only difference now is that the PGO build process no longer ignores the non-zero exit code of the test runner.
3. From the linked CPython issue It looks most likely that the problem is a test issue rather than an issue with Python's xml stdlib functionality itself, so it's safe to just skip the failing tests.

[edmorley]

Updated:
https://devcenter.heroku.com/articles/python-support#supported-runtimes
https://devcenter.heroku.com/articles/python-runtimes

Posted:
https://devcenter.heroku.com/changelog-items/3002

Closed out the roadmap item:
heroku/roadmap#316