Skip to content

Conversation

@cmaloney
Copy link
Contributor

@cmaloney cmaloney commented Nov 13, 2025

Memory usage now matches that of _io for large files.

$ ./python -Wd -I -m test test_io.test_largefile -uall,largefile,walltime -M32G -v
== CPython 3.15.0a1+ (heads/gh129005_take_bytes-dirty:d8b88498ebf, Nov 13 2025, 15:43:51) [Clang 21.1.5 ]
== Linux-6.17.7-arch1-1-x86_64-with-glibc2.42 little-endian
== Python build: debug
== cwd: <build_dir>/build/test_python_worker_116910æ
== CPU count: 32
== encodings: locale=UTF-8 FS=utf-8
== resources: all

Using random seed: 365280675
0:00:00 load avg: 9.60 Run 1 test sequentially in a single process
0:00:00 load avg: 9.60 [1/1] test_io.test_largefile
test_large_read (test.test_io.test_largefile.CLargeFileTest.test_large_read) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
ok
test_lseek (test.test_io.test_largefile.CLargeFileTest.test_lseek) ... ok
test_osstat (test.test_io.test_largefile.CLargeFileTest.test_osstat) ... ok
test_seek_read (test.test_io.test_largefile.CLargeFileTest.test_seek_read) ... ok
test_seek_readall (test.test_io.test_largefile.CLargeFileTest.test_seek_readall) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
 ... process data size: 2.3G
ok
test_seekable (test.test_io.test_largefile.CLargeFileTest.test_seekable) ... ok
test_truncate (test.test_io.test_largefile.CLargeFileTest.test_truncate) ... ok
test_large_read (test.test_io.test_largefile.PyLargeFileTest.test_large_read) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
ok
test_lseek (test.test_io.test_largefile.PyLargeFileTest.test_lseek) ... ok
test_osstat (test.test_io.test_largefile.PyLargeFileTest.test_osstat) ... ok
test_seek_read (test.test_io.test_largefile.PyLargeFileTest.test_seek_read) ... ok
test_seek_readall (test.test_io.test_largefile.PyLargeFileTest.test_seek_readall) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
 ... process data size: 2.3G
ok
test_seekable (test.test_io.test_largefile.PyLargeFileTest.test_seekable) ... ok
test_truncate (test.test_io.test_largefile.PyLargeFileTest.test_truncate) ... ok
test_it (test.test_io.test_largefile.TestCopyfile.test_it) ... ok
test_it (test.test_io.test_largefile.TestSocketSendfile.test_it) ... ok

----------------------------------------------------------------------
Ran 16 tests in 6.950s

OK
0:00:06 load avg: 9.72 [1/1] test_io.test_largefile passed

== Tests result: SUCCESS ==

1 test OK.

Total duration: 7.0 sec
Total tests: run=16
Total test files: run=1/1
Result: SUCCESS

For reference; this test goes from 8.5sec overall duration on my AMD 64 bit Arch Linux box to 7 seconds.

Memory usage now matches that of _io for large files
@cmaloney
Copy link
Contributor Author

Optimization enabled by gh-139871 (Trying to keep that one issue referencing all the things which I use it for; planning to expand the What's New with a single "here's all the modules that got faster" list once optimize)

@cmaloney cmaloney changed the title gh-129005: Remove copy from _pyio using take_bytes gh-129005: Remove copies from _pyio using take_bytes Nov 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant