ResolutionTooDeep while installing a set of packages, undebuggable

[wfjsw]

Description

Users are facing with different levels of backtracking depending on the packages (caused by extensions) they already installed. Sometimes there's resolution, but others got infinite backtrack and, finally, pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2000000

Related: #11480

cc @chflame163

Expected behavior

No response

pip version

23.0.1

Python version

3.10.11

OS

Windows

How to Reproduce

I can reproduce backtrack but not ResolutionTooDeep as I don't have the same set of installed packages. The log attached below is an instance of ResolutionTooDeep that may aid reproduction:

1. Download https://github.com/chflame163/ComfyUI_LayerStyle/blob/50c7862dc3337a0c11cebb2eae419a03c520bb10/requirements.txt
2. Install it.

Output

3b28df3d-2ffd-4509-8e18-5df67bd3753a.tmp.txt

Note: https://pypi.doubanio.com/simple is an up-to-date mirror of pypi. https://mirror.sjtu.edu.cn/pytorch-wheels/torch_stable.html is an outdated PyTorch mirror but it is not involved in actual resolution process. https://pypi.oystermercury.top/ms contains some wheels not available on pypi but is also not related.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[chflame163]

Someone has already reported this issue. I also really want to know how to solve it.
This issue seems to only occur when installing dependencies on the Aki ComfyUI Launcher. I tried manually installing requirements.txt with ComfyUI official portable or conda environment and did not encounter this issue.

[wfjsw]

Someone has already reported this issue. I also really want to know how to solve it. This issue seems to only occur when installing dependencies on the Aki ComfyUI Launcher. I tried manually installing requirements.txt with ComfyUI official portable or conda environment and did not encounter this issue.

As discussed with @notatallshaw at PyPA Discord, this seems to have something to do with already installed packages, where pip's attempt to use already satisfied packages significantly increases the burden at backtracking and in the end caused it to give up.

[notatallshaw]

Yeah, I'm looking to see if I can reproduce @wfjsw issue but in the mean time I suggest installing into an empty virtual environment, providing all your requirements upfront if you can.

[notatallshaw]

I can reproduce the original issue. (tl;dr the easiest workaround is to add --upgrade on to your pip install)

Steps to Reproduce

Using the environment pip 23.0.1, Windows, Python 3.10

1. Create an env.txt with the following contents:

numpy==1.26.4
pillow==10.4.0
matplotlib==3.8.0
Scipy==1.12.0
scikit_image==0.20.0
opencv-contrib-python==4.8.1.78
pymatting==1.1.12
segment_anything==1.0
timm==0.6.13
addict==2.4.0
yapf==0.40.2
colour-science==0.4.4
wget==3.2
mediapipe==0.10.7
loguru==0.7.2
typer_config==1.4.0
fastapi==0.111.1
rich==13.7.1
google-generativeai==0.7.2
diffusers==0.29.2
omegaconf==2.3.0
tqdm==4.66.4
transformers==4.43.3
kornia==0.7.1
blend_modes==2.1.0
blind-watermark==0.4.4
qrcode==7.4.2
pyzbar==0.1.9
transparent-background==1.3.1
huggingface_hub==0.24.2
accelerate==0.31.0
onnxruntime==1.17.0
psd-tools==1.9.34
fonttools==4.43.1
python-dateutil==2.8.2
pyparsing==3.1.1
packaging==24.1
cycler==0.12.1
kiwisolver==1.4.5
contourpy==1.1.1
tifffile==2023.9.26
lazy_loader==0.3
PyWavelets==1.5.0
networkx==3.1
imageio==2.31.5
numba==0.58.1
pyyaml==6.0.1
platformdirs==3.11.0
importlib-metadata==6.8.0
tomli==2.0.1
typing-extensions==4.8.0
flatbuffers==23.5.26
attrs==23.1.0
sounddevice==0.4.6
absl-py==2.0.0
protobuf==3.20.3
win32-setctime==1.1.0
colorama==0.4.6
typer==0.12.3
email_validator==2.2.0
starlette==0.37.2
fastapi-cli==0.0.4
httpx==0.26.0
python-multipart==0.0.9
uvicorn==0.30.3
jinja2==3.1.2
pydantic==2.8.2
pygments==2.18.0
markdown-it-py==3.0.0
google-ai-generativelanguage==0.6.6
google-api-core==2.19.1
google-api-python-client==2.138.0
google-auth==2.32.0
proto-plus==1.24.0
safetensors==0.4.2
filelock==3.12.4
requests==2.32.3
regex==2023.8.8
antlr4-python3-runtime==4.9.3
tokenizers==0.19.1
psutil==5.9.5
pandas==2.1.3
seaborn==0.13.0
opencv-python==4.8.1.78
py-cpuinfo==9.0.0
pypng==0.20220715.0
gdown==5.2.0
easydict==1.11
flet==0.23.2
albumentations==1.3.1
fsspec==2023.9.1
coloredlogs==15.0.1
sympy==1.12
gitpython==3.1.40
setuptools==65.5.0
click==8.1.7
aggdraw==1.3.18.post0
docopt==0.6.2
urllib3==1.26.18
pywin32==306
jsonschema==4.20.0
tabulate==0.9.0
python-dotenv==1.0.1
PrettyTable==3.9.0
cachetools==5.4.0
cryptography==43.0.0
boto3==1.34.150
botocore==1.34.150
shapely==2.0.2
openai==1.37.1
piexif==1.1.3
dill==0.3.8
ftfy==6.1.1
aiosignal==1.3.1
yarl==1.9.2
frozenlist==1.4.0
multidict==6.0.4
async-timeout==4.0.3
qudida==0.0.4
opencv-python-headless==4.7.0.72
sniffio==1.3.0
anyio==4.2.0
distro==1.9.0
pytz==2023.3.post1
six==1.16.0
jmespath==1.0.1
s3transfer==0.10.2
dnspython==2.6.1
idna==3.4

2. Run the command pip install -r env.txt --no-deps
3. Create a requirements.txt with the following contents:

numpy<2.0
pillow
torch
matplotlib
Scipy
scikit_image
scikit_learn
opencv-contrib-python
pymatting
segment_anything
timm
addict
yapf
colour-science
wget
mediapipe
loguru
typer_config
fastapi
rich
google-generativeai
diffusers
omegaconf
tqdm
transformers>=4.43.3
kornia
ultralytics>=8.2.0
blend_modes
blind-watermark
qrcode
pyzbar
transparent-background
huggingface_hub>=0.23.3
accelerate
onnxruntime
bitsandbytes>=0.41.1
torchscale
wandb
psd-tools
hydra-core
inference-cli>=0.13.0
inference-gpu[yolo-world]>=0.13.0

4. Run the command pip install -r requirements.txt

Broken Environments and Best Practises

Your original environment was actually "broken", i.e. if you ran pip check you would see you had packages installed that conflict with each other, that is, at least one package version installed which a different package says that version shouldn't be installed.

This is an unfortunate consequence of the fact that pip is "just an installer", it doesn't consider your whole environment and all your previous requests, it just installs what you tell it to install. This leads to the behaviour that starting with pip install {package1} followed by pip install {package2} can lead to a broken environment where as starting with pip install {package1} {package2} will never lead to a broken environment.

So, one way to keep things always working is to keep a requirements.txt with all your requirements you need, add to it every time you need something new, and always run pip install -r requirements.txt and because it has everything you need the chances of it breaking the environment is very low. If you do run into a problem you can always create a new virtual environment and install from that file, and because it has everything you need you're set.

Workarounds in your current situation

Normally the best things to do it upgrade your version of pip, unfortunately it looks like this scenario hits a bug in the latest version of pip: #12768 / #12317. I need to do some more testing, but I strongly suspect once a new version of resolvelib is released (sarugaku/resolvelib#159) and pip upgrades to that new version of resolvelib, this will be fixed in pip. I do not have an ETA for that yet.

One workaround you can quickly do in the future is to use --upgrade, i.e. at step 4. do pip install -r requirements.txt --upgrade, this tells pip not to prefer the already installed versions of packages, and to get the latest versions based on your requirements, in the above example this causes pip to be able to install the packages fine. In general, this is probably a good options when your environment is broken.

Another workaround is to use uv, a competitor to pip that has a "pip like" interface, i.e. you run uv pip install instead of pip install. They are focused on being hyper fast and can probably do any kind of resolution you need, they are not 100% compatible with pip, but for any modern package it should work. Installation guide: https://docs.astral.sh/uv/getting-started/installation/. You will probably need to read the pip compatability guide particularly the section "Packages that exist on multiple indexes": https://docs.astral.sh/uv/pip/compatibility

[notatallshaw]

Development note: A solution to the problem of the resolver getting stuck because it is trying to use the users existing packages and those packages are already conflicting might be to prefer already installed packages during backtracking, this should quickly solve that the current packages installed need new non-conflicting versions. I am unsure if the required information is available during resolution, but it's worth looking at after that next resolvelib vendoring.

[notatallshaw]

I did a bit more testing on this today, following the scenario I outlined in #12972 (comment):

• pip 23.0.1: ResolutionTooDeep
• pip 24.2 / main: ResolutionImpossible (incorrect)
• pip main with resolvelib main vendored: ResolutionTooDeep 🙁

So there is still work to do once resolvelib has been vendored, I will use this as a test case for improvements to resolution.

[wfjsw]

Hmm... Very counterintuitively, there is almost nothing I can do to prevent a broken environment, similarly in a way to survive a constant dependency hell with a pile of overstrict or ill-defined list of specification across hundreds of libraries. This is well out of the scope of pip but still a cold reality. 😟

I'm worried about --upgrade changing too many out of scope packages. There is a certain list of things I never want it to touch on automatically, for example torch (it will always install the wrong version) and onnxruntime (the installation order matters💩). I'm in halfways researching for a pinning method that would not fail the resolution process (more literally broken environment but actually works). Similar issue for uv.

Let me check what uv has to provide.

[notatallshaw]

Very counterintuitively, there is almost nothing I can do to prevent a broken environment

You can just project / environment management tools, like pdm, uv, rye, or poetry. Instead of installing a set at a time you "add" to your project, e.g. "uv add requests", they then keep track of your environment and make sure it doesn't get broken. And I know that uv and pdm both have options to override specifications when they are not correct.

I'm worried about --upgrade changing too many out of scope packages. There is a certain list of things I never want it to touch on automatically, for example torch (it will always install the wrong version) and onnxruntime (the installation order matters💩).

You should pin these tools to the exact version you want for those packages then, either by adding the exact version with a project / environment management tool, or pinning the exact version you want with a constraints file with pip.

I appreciate this is all a learning curve, and managing a Python environment isn't easy as libraries depend on each other in complex ways. Unfortunately pip is extremely limited in offering anything other than install options here, it was designed in a world when the Python ecosystem wasn't so complex, and it doesn't have the resources to expand into project and environment management.

Pip should eventually be able to install the commands as listed, as long as I can keep working on resolution, but it might be several releases before this particular scenario works as change is slow due to pip being depended on by so many users and resources being so limited.