New resolver takes 1-2 hours to install a large requirements file

[bdraco]

Description

This is a followup to #10788

Attached is a callgrind
pip_callgrind.zip

The bulk of the time is spent creating Version objects

Expected behavior

No response

pip version

pip 23.2.1 from /root/ztest/lib/python3.11/site-packages/pip (python 3.11)

Python version

3.11

OS

alpine 3.18 linux

How to Reproduce

Must use alpine 3.18 linux

python3 -m cProfile -o all.pstats -m pip install --no-cache-dir --only-binary=:all: --index-url https://wheels.home-assistant.io/musllinux-index/ -r /usr/src/homeassistant/requirements_all.txt

https://github.com/home-assistant/core/blob/dev/requirements_all.txt

or

Dockerfile https://github.com/home-assistant/core/blob/dev/Dockerfile

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

Hey, I'll take a look at this in more detail tonight but just quickly:

The bulk of the time is spent creating Version objects

If I'm reading your graph correctly 9% of the time is spent on version objects? So, if I understand correctly, that's not the same as creating and not really the bulk of the time. But feel to correct me.

Edit: I see you already have a PR that proves your point, exciting! Please ignore everything I said 🙂

[bdraco]

Hey, I'll take a look at this in more detail tonight but just quickly:

The bulk of the time is spent creating Version objects

If I'm reading your graph correctly 9% of the time is spent on version objects? So, if I understand correctly, that's not the same as creating and not really the bulk of the time. But feel to correct me.

Edit: I see you already have a PR that proves your point, exciting! Please ignore everything I said 🙂

I was doing profiling on my laptop which means the ssl read / network latency overhead is much higher than what actually happens in production. The top of the profile calls to read is outsized compared to what actually happens on github actions when Home Assistant is building images. Since thats not within the control of pip and will vary greatly based on the network, I did not factor it in it for the purposes of comparing profiles.

[notatallshaw]

Well I got home and tried to reproduce this (had a lot of fun learning how to install Python 3.11 into an Alpine WSL 2 distro) but I could not produce a significant performance improvement with your branch as you suggest.

Some notes:

1. I used "install --dry-run" to remove wheel install from timing
2. I let one install run first to build up cache to reduce any network variance
3. I compared pypa-pip/main with bdraco-pip/version_cache

So first install pip/main and build cache

python -m pip install git+https://github.com/pypa/pip.git@main
python -m pip install --dry-run --only-binary=:all: --index-url https://wheels.home-assistant.io/musllinux-index/ -r requirements_all.txt

And then collect timing:

time python -m pip install --dry-run --only-binary=:all: --index-url https://wheels.home-assistant.io/musllinux-index/ -r requirements_all.txt

Which produced:

real    7m 51.29s
user    0m 45.65s
sys     0m 2.49s

And then installing your branch and collect timing:

python -m pip install git+https://github.com/bdraco/pip.git@version_cache
time python -m pip install --dry-run --only-binary=:all: --index-url https://wheels.home-assistant.io/musllinux-index/ -r requirements_all.txt

Which produced:

real    7m 44.08s
user    0m 45.00s
sys     0m 2.73s

Which potentially is a real performance gain, but it seems less than 2%.

Can I suggest you test again on your side using pip/main instead of pip/23.2.1, perhaps the performance difference is something that already landed? Or maybe a problem in the way alpine has packaged/vendored pip?

[bdraco]

Thanks for testing. I think the major difference is going to be that we are running out image builds for arm under qemu which is much more performance sensitive.

[notatallshaw]

Thanks for testing. I think the major difference is going to be that we are running out image builds for arm under qemu which is much more performance sensitive.

But why just the version object creation? Wouldn't being on a worse performane machine cause everything to be slower? Have you tried running vanilla pip/main with no profiling enabled?

As a side note if performance is so critical to you here you might want to try https://github.com/prefix-dev/rip once it is released. I tried testing now but getting it to work under Alpine defeated me.

[bdraco]

It would probably be helpful to see an actual production run : https://github.com/home-assistant/core/actions/runs/6408628446/job/17398078969

But why just the version object creation?

It was the thing that made the most difference in testing

Wouldn't being on a worse performane machine cause everything to be slower?

Yes, everything will be worse

Have you tried running vanilla pip/main with no profiling enabled?

Yes

As a side note if performance is so critical to you here you might want to try prefix-dev/rip once it is released. I tried testing now but getting it to work under Alpine defeated me.

Thanks for that.

[bdraco]

I'm going to blow away my test env, and try again with your testing steps above.

[bdraco]

I'm going to try the following.

Create a fresh venv, and install the packages

python -m pip install git+https://github.com/pypa/pip.git@main
python -m pip install --no-cache-dir --only-binary=:all: --index-url https://wheels.home-assistant.io/musllinux-index/ -r /usr/src/homeassistant/requirements.txt  -r /usr/src/homeassistant/requirements_all.txt 

Once the packages are in the venv, just do the resolution with:

python -m pip install git+https://github.com/pypa/pip.git@main
time python -m pip install --dry-run --no-cache-dir --only-binary=:all: --index-url https://wheels.home-assistant.io/musllinux-index/ -r /usr/src/homeassistant/requirements.txt  -r /usr/src/homeassistant/requirements_all.txt 

python -m pip install git+https://github.com/bdraco/pip.git@version_cache
time python -m pip install --dry-run --no-cache-dir --only-binary=:all: --index-url https://wheels.home-assistant.io/musllinux-index/ -r /usr/src/homeassistant/requirements.txt  -r /usr/src/homeassistant/requirements_all.txt 

[bdraco]

One thing that stands out is the image builds run with --no-cache-dir

[bdraco]

With the above testing version_cache is slower 🤦 ... will make sure its actually being used next

main once all packages installed:

real	0m59.382s
user	0m58.529s
sys	0m0.688s

version_cache once all packages installed:

real	1m0.171s
user	0m58.899s
sys	0m0.739s

[bdraco]

So it looks like the pip install isn't actually working ./lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py is not the version_cache file