Resolving Home Assistant takes 12 hours with new resolver

[balloob]

Description

The new resolver introduced in pip takes 12 hours to resolve Home Assistant, making it unusable for Home Assistant. Our current workaround is using the legacy resolver.

(Time measured on AMD Ryzen 7, 32GB memory)

The legacy resolver is going to be removed in #9631 making pip unusable for Home Assistant.

Home Assistant focuses on privacy and local control and is world’s largest open source home automation platform. In 2020 it was the 2nd most active Python project on GitHub (per State of the Octoverse 2020). We integrate over 1000 different APIs and each API is integrated via a Python package installed via pip (requirements.txt).

All our dependency versions are pinned, but the libraries that we use don’t pin their dependencies. This leads to long resolving time as the new resolver will download all the different potentially compatible versions to find the right one. Libraries don’t pin their dependencies because a library tries to be compatible with as many other projects as possible (we always make sure we don’t rely on HA specific API implementations).

I understand you don’t want to keep the legacy resolver around forever, but we would like to see that the new resolver is usable prior to it’s removal.

Expected behavior

Done in less than 15 minutes.

pip version

21.3.1

Python version

3.10

OS

Debian 11

How to Reproduce

git clone https://www.github.com/home-assistant/core home-assistant-core
cd home-assistant-core
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements_all.txt

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

I can not reproduce, I get ResolutionImpossible pretty quickly (bar download time).

I tested on Ubuntu 20.04.3 LTS and changed the commands a little bit so they could be easily repeated without causing the state of the environment to change:

git clone https://www.github.com/home-assistant/core home-assistant-core
cd home-assistant-core
python3.10 -m venv .venv
source .venv/bin/activate
python -m pip install pip --upgrade
python -m pip download -r requirements_all.txt -d downloads

The ResolutionImpossible seems to related to aiohttp which is set to 3.8.1 in the top level requirements and constraints file but at least one of the requirements discord-py specifies it as aiohttp<3.8.0 and >=3.6.0. Full conflict error I get:

The conflict is caused by:
    The user requested aiohttp==3.8.1
    adax-local 0.1.3 depends on aiohttp>=3.0.6
    accuweather 0.3.0 depends on aiohttp
    adax 0.2.0 depends on aiohttp>=3.0.6
    adguardhome 0.5.0 depends on aiohttp>=3.0.0
    advantage-air 0.2.5 depends on aiohttp
    afsapi 0.0.4 depends on aiohttp>=3.3.2
    aio-geojson-geonetnz-quakes 0.13 depends on aiohttp<4 and >=3.7.4
    aio-geojson-geonetnz-volcano 0.6 depends on aiohttp<4 and >=3.7.4
    aio-geojson-nsw-rfs-incidents 0.4 depends on aiohttp<4 and >=3.7.4
    aioambient 2021.11.0 depends on aiohttp<4.0.0 and >=3.7.4
    aioaseko 0.0.1 depends on aiohttp
    aioazuredevops 1.3.5 depends on aiohttp>=3.6.2
    aiobotocore 1.2.2 depends on aiohttp>=3.3.1
    aioeafm 0.1.2 depends on aiohttp<4.0.0 and >=3.6.1
    aioeagle 1.1.0 depends on aiohttp
    aioemonitor 1.0.5 depends on aiohttp>=3.7.0
    aioflo 2021.11.0 depends on aiohttp<4.0.0 and >=3.7.4
    aiogithubapi 21.11.0 depends on aiohttp<4.0 and >=3.6.1
    aioguardian 2021.11.0 depends on aiohttp<4.0.0 and >=3.7.4post0
    aioharmony 0.2.9 depends on aiohttp
    aiohttp-cors 0.7.0 depends on aiohttp>=1.1
    aiohue 3.0.11 depends on aiohttp
    aiohwenergy 0.6.0 depends on aiohttp
    aiolookin 0.1.0 depends on aiohttp>=3.7.4
    aiolyric 1.0.8 depends on aiohttp>=3.7.3
    aiomodernforms 0.1.8 depends on aiohttp>=3.0.0
    aiomusiccast 0.14.3 depends on aiohttp<4.0.0 and >=3.7.4
    aionanoleaf 0.1.1 depends on aiohttp
    aionotion 3.0.2 depends on aiohttp<4.0.0 and >=3.7.4
    aiopvapi 1.6.19 depends on aiohttp<4 and >=3.7.4
    aiopvpc 3.0.0 depends on aiohttp>=3.7.4.post0
    aiorecollect 1.0.8 depends on aiohttp<4.0.0 and >=3.7.4
    aioridwell 2021.12.2 depends on aiohttp>=3.8.0
    aioshelly 1.0.7 depends on aiohttp
    aiosteamist 0.3.1 depends on aiohttp<4.0.0 and >=3.8.1
    aiosyncthing 0.5.1 depends on aiohttp>=3.7.4
    aiotractive 0.5.2 depends on aiohttp>=3.7.4
    aiounifi 29 depends on aiohttp
    aiowatttime 0.1.1 depends on aiohttp<4.0.0 and >=3.7.4
    aioymaps 1.2.2 depends on aiohttp>=3.0.0
    airly 1.1.0 depends on aiohttp>=3.5.4
    airthings-cloud 0.1.0 depends on aiohttp>=3.0.6
    alpha-vantage 2.3.1 depends on aiohttp
    ambee 0.4.0 depends on aiohttp>=3.0.0
    ambiclimate 0.2.1 depends on aiohttp>=3.0.6
    asmog 0.0.6 depends on aiohttp
    async-upnp-client 0.23.3 depends on aiohttp>=3.7.4
    asyncpysupla 0.0.5 depends on aiohttp
    blebox-uniapi 1.3.3 depends on aiohttp>=3
    bond-api 0.1.15 depends on aiohttp>=3.6.1
    brunt 1.1.1 depends on aiohttp
    bsblan 0.4.0 depends on aiohttp>=3.0.0
    connect-box 0.2.8 depends on aiohttp
    coronavirus 1.1.1 depends on aiohttp>=3.0.0
    crownstone-cloud 1.4.9 depends on aiohttp~=3.7
    crownstone-sse 2.0.3 depends on aiohttp~=3.7
    directv 0.4.0 depends on aiohttp>=3.6.2
    discord-py 1.7.3 depends on aiohttp<3.8.0 and >=3.6.0
    The user requested (constraint) aiohttp==3.8.1

[pradyunsg]

Thanks for filing this @balloob! ^>^

While I haven't started using Home Assistant yet, it's been on my radar and I appreciate the work that you and other volunteers who have worked on it, have put into it!

Much like @notatallshaw, I'm unable to reproduce this with pip 21.3.1 on MacOS, with a ResolutionImpossible error after 4m 4s with unpopulated cache (i.e. everything had to download). With a populated cache, that takes 2m 0s.

The error is that exact same as @notatallshaw, as far as I can tell.

The conflict is caused by:
    The user requested aiohttp==3.8.1
    adax-local 0.1.3 depends on aiohttp>=3.0.6
    accuweather 0.3.0 depends on aiohttp
    adax 0.2.0 depends on aiohttp>=3.0.6
    adguardhome 0.5.0 depends on aiohttp>=3.0.0
    advantage-air 0.2.5 depends on aiohttp
    afsapi 0.0.4 depends on aiohttp>=3.3.2
    aio-geojson-geonetnz-quakes 0.13 depends on aiohttp<4 and >=3.7.4
    aio-geojson-geonetnz-volcano 0.6 depends on aiohttp<4 and >=3.7.4
    aio-geojson-nsw-rfs-incidents 0.4 depends on aiohttp<4 and >=3.7.4
    aioambient 2021.11.0 depends on aiohttp<4.0.0 and >=3.7.4
    aioaseko 0.0.1 depends on aiohttp
    aioazuredevops 1.3.5 depends on aiohttp>=3.6.2
    aiobotocore 1.2.2 depends on aiohttp>=3.3.1
    aioeafm 0.1.2 depends on aiohttp<4.0.0 and >=3.6.1
    aioeagle 1.1.0 depends on aiohttp
    aioemonitor 1.0.5 depends on aiohttp>=3.7.0
    aioflo 2021.11.0 depends on aiohttp<4.0.0 and >=3.7.4
    aiogithubapi 21.11.0 depends on aiohttp<4.0 and >=3.6.1
    aioguardian 2021.11.0 depends on aiohttp<4.0.0 and >=3.7.4post0
    aioharmony 0.2.9 depends on aiohttp
    aiohttp-cors 0.7.0 depends on aiohttp>=1.1
    aiohue 3.0.11 depends on aiohttp
    aiohwenergy 0.6.0 depends on aiohttp
    aiolookin 0.1.0 depends on aiohttp>=3.7.4
    aiolyric 1.0.8 depends on aiohttp>=3.7.3
    aiomodernforms 0.1.8 depends on aiohttp>=3.0.0
    aiomusiccast 0.14.3 depends on aiohttp<4.0.0 and >=3.7.4
    aionanoleaf 0.1.1 depends on aiohttp
    aionotion 3.0.2 depends on aiohttp<4.0.0 and >=3.7.4
    aiopvapi 1.6.19 depends on aiohttp<4 and >=3.7.4
    aiopvpc 3.0.0 depends on aiohttp>=3.7.4.post0
    aiorecollect 1.0.8 depends on aiohttp<4.0.0 and >=3.7.4
    aioridwell 2021.12.2 depends on aiohttp>=3.8.0
    aioshelly 1.0.7 depends on aiohttp
    aiosteamist 0.3.1 depends on aiohttp<4.0.0 and >=3.8.1
    aiosyncthing 0.5.1 depends on aiohttp>=3.7.4
    aiotractive 0.5.2 depends on aiohttp>=3.7.4
    aiounifi 29 depends on aiohttp
    aiowatttime 0.1.1 depends on aiohttp<4.0.0 and >=3.7.4
    aioymaps 1.2.2 depends on aiohttp>=3.0.0
    airly 1.1.0 depends on aiohttp>=3.5.4
    airthings-cloud 0.1.0 depends on aiohttp>=3.0.6
    alpha-vantage 2.3.1 depends on aiohttp
    ambee 0.4.0 depends on aiohttp>=3.0.0
    ambiclimate 0.2.1 depends on aiohttp>=3.0.6
    asmog 0.0.6 depends on aiohttp
    async-upnp-client 0.23.3 depends on aiohttp>=3.7.4
    asyncpysupla 0.0.5 depends on aiohttp
    blebox-uniapi 1.3.3 depends on aiohttp>=3
    bond-api 0.1.15 depends on aiohttp>=3.6.1
    brunt 1.1.1 depends on aiohttp
    bsblan 0.4.0 depends on aiohttp>=3.0.0
    connect-box 0.2.8 depends on aiohttp
    coronavirus 1.1.1 depends on aiohttp>=3.0.0
    crownstone-cloud 1.4.9 depends on aiohttp~=3.7
    crownstone-sse 2.0.3 depends on aiohttp~=3.7
    directv 0.4.0 depends on aiohttp>=3.6.2
    discord-py 1.7.3 depends on aiohttp<3.8.0 and >=3.6.0
    The user requested (constraint) aiohttp==3.8.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

You are likely getting a warning from the legacy resolver about dependency conflicts as well -- it just does not fail when it encounters a dependency conflict.

[notatallshaw]

I experimented to see if I could "fix" the ResolutionImpossible, I tried slowly loosening the requirements to allow pip to find compatible versions. This revealed a lot of conflicting requirements based on aiohttp, websockets, click, boto3, etc..

Here are two cases I found which seemed unresolvable:

1. mycroftapi==2.0 depends on the very old websocket-client==0.44.0. But mycroftapi but it hasn't been updated in over 3 years and the home page link returns a 404: https://pypi.org/project/mycroftapi/ . Rather than trying to find very old versions of the 4 projects that require newer than websocket-client==0.44.0 I just dropped the mycroftapi requirement.
2. pysmarty 0.8 requires pymodbus==1.5.2 and pystiebeleltron 0.0.1.dev2 requires pymodbus>=2.1.0. This doesn't seem resolvable as pysmarty doesn't seem to be updated any more (no update in 2-3 years) and pystiebeleltron only has 2 non-release versions and no updates in over 1 year. I drop[ed pystiebeleltron from the requirements as it is an unreleased version.

I continued loosening requirements but unfortunately this eventually led to very long backtracking times. I used an experimental version of Pip I have which implements backjumping to try and speed up these very complex scenarios and continued trying to loosen the requirements.

Eventually I was able to find a set of requirements that didn't conflict (minus the two packages above), and after switching to Python 3.9 to build them (I could not get homeassistant-pyozw to build on 3.10). Here is the output of pip freeze, perhaps you can compare it to a pip freeze in a working environment you have, and update them 1 at a time to see what causes conflicts:

Pip Freeze Output

abodepy==1.2.0
accuweather==0.3.0
acme==1.22.0
Adafruit-Blinka==5.5.1
adafruit-circuitpython-bmp280==3.1.1
adafruit-circuitpython-busdevice==5.1.1
adafruit-circuitpython-dht==3.7.0
adafruit-circuitpython-mcp230xx==2.2.2
adafruit-circuitpython-pca9685==3.3.2
adafruit-circuitpython-register==1.9.7
Adafruit-GPIO==1.0.3
Adafruit-PlatformDetect==3.19.2
Adafruit-PureIO==1.1.9
Adafruit-SHT31==1.0.2
adal==1.2.7
adax==0.2.0
Adax-local==0.1.3
adb-shell==0.4.0
adext==0.4.2
adguardhome==0.5.0
advantage-air==0.2.5
AEMET-OpenData==0.2.1
aenum==3.1.5
afsapi==0.0.4
agent-py==0.0.23
aio-geojson-client==0.15
aio-geojson-geonetnz-quakes==0.13
aio-geojson-geonetnz-volcano==0.6
aio-geojson-nsw-rfs-incidents==0.4
aio-georss-client==0.8
aio-georss-gdacs==0.5
aioambient==2.0.0
aioaseko==0.0.1
aioasuswrt==1.4.0
aioazuredevops==1.3.5
aiobotocore==2.1.0
aiocache==0.11.1
aiocoap==0.4.3
aioconsole==0.3.3
aiodiscover==1.4.5
aiodns==3.0.0
aioeafm==0.1.2
aioeagle==1.1.0
aioemonitor==1.0.5
aioesphomeapi==10.6.0
aiofiles==0.8.0
aioflo==2021.11.0
aioftp==0.12.0
aiogithubapi==21.11.0
aioguardian==2021.11.0
aioharmony==0.2.9
aiohomekit==0.6.10
aiohttp==3.7.4.post0
aiohttp-cors==0.7.0
aiohue==3.0.11
aiohwenergy==0.6.0
aioimaplib==0.9.0
aioitertools==0.8.0
aiokafka==0.6.0
aiokef==0.2.16
aiolifx==0.7.0
aiolifx-effects==0.2.2
aiolookin==0.1.0
aiolyric==1.0.8
aiomodernforms==0.1.8
aiomusiccast==0.14.3
aionanoleaf==0.1.1
aionotify==0.2.0
aionotion==3.0.2
aiooncue==0.3.0
aiopulse==0.4.3
aiopvapi==1.6.19
aiopvpc==3.0.0
aiopylgtv==0.4.0
aiorecollect==1.0.8
aioridwell==2021.10.0
aiosenseme==0.5.5
aioserial==1.3.0
aioshelly==1.0.7
aioshutil==1.1
aiosqlite==0.17.0
aiosteamist==0.1.0
aioswitcher==2.0.6
aiosyncthing==0.5.1
aiotractive==0.5.2
aiounifi==29
aiovlc==0.1.0
aiowatttime==0.1.1
aioymaps==1.2.2
airly==1.1.0
airthings-cloud==0.1.0
airtouch4pyapi==1.0.5
alabaster==0.7.12
aladdin-connect==0.3
alarmdecoder==1.13.11
alpha-vantage==2.3.1
ambee==0.4.0
amberelectric==1.0.3
Ambiclimate==0.2.1
amcrest==1.9.3
android-backup==0.2.0
androidtv==0.0.60
anel-pwrctrl-homeassistant==0.0.1.dev2
anthemav==1.2.0
anyio==3.5.0
apcaccess==0.0.13
apns2==0.3.0
appdirs==1.4.4
apprise==0.9.6
aprslib==0.6.46
APScheduler==3.6.3
aqualogic==2.6
arcam-fmj==0.12.0
argcomplete==2.0.0
arris-tg2492lg==1.1.0
arrow==1.2.1
asmog==0.0.6
asn1==2.4.2
asterisk-mbox==0.5.0
astral==2.2
astroid==2.9.3
async-timeout==3.0.1
async-upnp-client==0.23.3
asyncclick==7.1.2.3
asynccmd==0.2.4
asyncio==3.4.3
asyncio-dgram==2.1.1
asyncio-mqtt==0.11.1
asyncio-throttle==1.0.2
asyncpysupla==0.0.5
asyncssh==2.8.1
asyncstdlib==3.10.2
asynctest==0.13.0
atenpdu==0.3.2
atomicwrites==1.4.0
attrs==21.2.0
auroranoaa==0.0.2
aurorapy==0.2.6
Authlib==0.15.5
av==8.1.0
awesomeversion==22.1.0
axis==44
azure-common==1.1.27
azure-core==1.21.1
azure-eventhub==5.5.0
azure-servicebus==0.50.3
Babel==2.9.1
backoff==1.11.1
baidu-aip==1.6.6.0
base36==0.1.1
batinfo==0.4.2
bcrypt==3.1.7
beautifulsoup4==4.10.0
bellows==0.29.0
bimmer-connected==0.8.7
bitarray==2.3.5
bitstring==3.1.9
bizkaibus==0.1.1
bleak==0.14.1
blebox-uniapi==1.3.3
blinkpy==0.18.0
BlinkStick==1.2.0
blockchain==1.4.4
blurhash==1.1.4
bond-api==0.1.15
boschshcpy==0.2.28
boto3==1.20.24
botocore==1.23.24
bravia-tv==1.0.11
broadlink==0.18.0
brother==1.1.0
brotlipy==0.7.0
brottsplatskartan==0.0.1
brunt==1.1.1
bs4==0.0.1
bsblan==0.4.0
bt-proximity==0.2.1
bthomehub5-devicelist==0.1.1
btlewrap==0.0.10
btsmarthub-devicelist==0.2.0
buienradar==1.0.5
cached-property==1.5.2
cachetools==4.2.4
caldav==0.8.2
casttube==0.2.1
cattrs==1.10.0
cbor2==5.4.2
cchardet==2.1.7
certifi==2021.10.8
cffi==1.15.0
cfgv==3.3.1
chardet==4.0.0
charset-normalizer==2.0.10
circuit-webhook==1.0.1
ciscomobilityexpress==0.3.9
ciso8601==2.2.0
clearpasspy==1.0.2
click==7.1.2
click-log==0.3.2
click-plugins==1.1.1
clx-sdk-xms==1.0.0
CO2Signal==0.4.2
coinbase==2.1.0
colorama==0.4.4
coloredlogs==15.0.1
colorlog==6.6.0
colorthief==0.2.1
colorzero==2.0
colour==0.1.5
commentjson==0.9.0
commonmark==0.9.1
concord232==0.15
config==0.5.1
ConfigArgParse==1.5.3
configparser==5.2.0
connect-box==0.2.8
construct==2.10.56
convertdate==2.3.2
coronavirus==1.1.1
coverage==6.2
crc16==0.1.1
crccheck==1.1
crcmod==1.7
croniter==1.0.6
crownstone-cloud==1.4.9
crownstone-core==3.1.0
crownstone-sse==2.0.3
crownstone-uart==2.1.0
cryptography==35.0.0
cssselect==1.1.0
cycler==0.11.0
Cython==0.29.26
dacite==1.6.0
dataclasses-json==0.5.3
datadog==0.15.0
datapoint==0.9.8
dateparser==1.1.0
DateTime==4.3
dbus-next==0.2.3
debugpy==1.5.1
decorator==5.1.1
deepmerge==1.0.1
defusedxml==0.7.1
deluge-client==1.7.1
demjson3==3.0.5
denonavr==0.10.9
Deprecated==1.2.13
devolo-home-control-api==0.17.4
devolo-plc-api==0.7.1
dicttoxml==1.7.4
directv==0.4.0
discogs-client==2.3.0
discord.py==1.7.3
discovery30303==0.2.1
distlib==0.3.4
dlipower==0.7.165
dnspython==2.1.0
docopt==0.6.2
docutils==0.17.1
DoorBirdPy==2.1.0
dovado==0.4.1
dsmr-parser==0.32
DTLSSocket==0.1.12
dwdwfsapi==1.0.4
dweepy==0.3.0
dynalite-devices==0.1.46
ebusdpy==0.0.17
ecdsa==0.17.0
ecoaliface==0.4.0
elgato==3.0.0
eliqonline==1.2.2
elkm1-lib==1.0.0
elmax-api==0.0.2
emoji==1.5.0
emulated-roku==0.2.1
enocean==0.50.0
enturclient==0.2.2
enum-compat==0.0.3
env-canada==0.5.20
envoy-reader==0.20.1
envoy-utils==0.0.1
envs==1.4
ephem==3.7.7.0
epson-projector==0.4.2
epsonprinter==0.0.9
esprima==4.0.1
eternalegypt==0.0.12
evohome-async==0.3.15
faadelays==0.0.7
fake-useragent==0.1.11
fastdotcom==0.0.3
feedparser==6.0.2
fiblary3==0.1.7
filelock==3.4.2
fints==1.0.1
fitbit==0.3.1
fixerio==1.0.0a0
fjaraskupan==1.0.2
Flask==2.0.2
flipr-api==1.4.1
flux-led==0.28.2
fnvhash==0.1.0
fonttools==4.28.5
foobot-async==1.0.0
forecast-solar==2.1.0
fortiosapi==1.0.5
freebox-api==0.0.10
freesms==0.2.0
fritzconnection==1.8.0
future==0.18.2
garages-amsterdam==3.0.0
gcloud==0.18.3
geniushub-client==0.6.30
geographiclib==1.52
geojson==2.5.0
geojson-client==0.6
geopy==2.1.0
georss-client==0.14
georss-generic-client==0.6
georss-ign-sismologia-client==0.3
georss-qld-bushfire-alert-client==0.5
get-mac==0.8.3
getmac==0.8.2
gios==2.1.0
gitterpy==0.1.7
glances-api==0.2.0
gntp==1.0.3
goalzero==0.2.1
goodwe==0.2.10
google-api-core==1.31.5
google-api-python-client==1.6.4
google-auth==1.35.0
google-auth-oauthlib==0.4.6
google-cloud-pubsub==2.9.0
google-cloud-texttospeech==0.4.0
google-nest-sdm==1.3.0
googleapis-common-protos==1.54.0
googlemaps==2.5.1
goslide-api==0.5.1
gpiozero==1.5.1
gps3==0.33.3
graphql-subscription-manager==0.4.3
graphviz==0.19.1
greeclimate==1.0.1
greeneye-monitor==3.0.1
greenlet==1.1.2
greenwavereality==0.5.1
growattServer==1.1.0
grpc-google-iam-v1==0.12.3
grpcio==1.43.0
gstreamer-player==1.1.2
gTTS==2.2.3
guppy3==3.1.2
h11==0.12.0
h2==2.6.2
ha-ffmpeg==3.0.2
ha-philipsjs==2.7.6
habitipy==0.2.0
hangups==0.4.17
HAP-python==4.4.0
hass-nabucasa==0.51.0
hass-splunk==0.1.1
HATasmota==0.3.1
haversine==2.5.1
hbmqtt==0.9.6
hdate==0.10.4
heatmiserV3==1.1.18
herepy==2.0.0
hijri-converter==2.2.2
hikvision==0.4
hkavr==0.0.5
hlk-sw16==0.0.9
hole==0.6.0
holidays==0.12
home-assistant-frontend==20211229.1
homeassistant-pyozw==0.1.10
homeconnect==0.6.3
homematicip==1.0.1
homepluscontrol==0.0.5
horimote==0.4.1
hpack==3.0.0
html-table-parser-python3==0.1.5
httmock==1.4.0
http-ece==1.1.0
httpcore==0.14.4
httplib2==0.19.0
httpsig==1.3.0
httpx==0.21.0
huawei-lte-api==1.4.18
huisbaasje-client==0.1.0
humanfriendly==10.0
Hydrawiser==0.2
hyper==0.7.0
hyperframe==3.2.0
hyperion-py==0.7.4
iammeter==0.1.7
iaqualink==0.4.1
ibm-cloud-sdk-core==3.13.2
ibm-watson==5.2.2
ibmiotf==0.3.4
icmplib==3.0
identify==2.4.3
idna==3.3
ifaddr==0.1.7
iglo==1.2.7
ihcsdk==2.7.0
imageio==2.13.5
imagesize==1.3.0
importlib-metadata==4.10.0
incomfort-client==0.4.4
inflection==0.5.1
influxdb==5.3.1
influxdb-client==1.24.0
iniconfig==1.1.1
intelhex==2.3.0
iotawattpy==0.1.0
iperf3==0.1.11
ismartgate==4.0.4
iso4217==1.7.20211001
iso8601==1.0.2
isodate==0.6.1
isort==5.10.1
itsdangerous==2.0.1
jellyfin-apiclient-python==1.7.2
Jinja2==3.0.3
jmespath==0.10.0
josepy==1.12.0
Js2Py==0.71
jsonpatch==1.32
jsonpath==0.82
jsonpickle==2.0.0
jsonpointer==2.2
jsonrpc-async==2.1.0
jsonrpc-base==2.1.0
jsonrpc-websocket==3.1.1
jsonschema==4.4.0
justbackoff==0.6.0
jwcrypto==1.0
kafka-python==2.0.2
kaiterra-async-client==0.0.2
keba-kecontact==1.1.0
kiwiki-client==0.1.1
kiwisolver==1.3.2
konnected==1.2.0
korean-lunar-calendar==0.2.1
kostal-plenticore==0.2.0
krakenex==2.1.0
lakeside==0.12
lark-parser==0.7.8
lazy-object-proxy==1.7.1
libcst==0.4.0
libpyfoscam==1.0
libpyvivotek==0.4.0
librouteros==3.2.0
libsoundtouch==0.8.0
life360==4.1.1
lightify==1.0.7.3
lightwave==0.19
limitlessled==1.1.3
linode-api==4.1.9b1
lmnotify==0.0.4
locationsharinglib==4.1.5
logi-circle==0.2.2
loguru==0.5.3
lomond==0.3.3
london-tube-status==0.2
luftdaten==0.7.1
lupupy==0.0.24
lw12==0.9.2
lxml==4.7.1
mac-vendor-lookup==0.1.11
magicseaweed==1.0.3
Markdown==3.3.6
MarkupSafe==2.0.1
marshmallow==3.14.1
marshmallow-dataclass==8.5.3
marshmallow-enum==1.5.1
Mastodon.py==1.5.1
matplotlib==3.5.1
matrix-client==0.4.0
maxcube-api==0.4.3
mbddns==0.1.2
mccabe==0.6.1
mcstatus==6.0.0
MechanicalSoup==0.12.0
mediafile==0.9.0
messagebird==1.2.0
metar==1.9.0
meteoalertapi==0.2.0
meteofrance-api==1.0.2
mficlient==0.3.0
micloud==0.5
miflora==0.7.0
mill-local==0.1.1
millheater==0.9.0
miniaudio==1.46
minio==5.0.10
mitemp-bt==0.0.5
mock==4.0.3
motionblinds==0.5.8
motioneye-client==0.3.12
ms-cv==0.1.1
msgpack==1.0.3
msrest==0.6.21
msrestazure==0.6.4
mt-940==4.23.0
mullvad-api==1.0.0
multidict==5.2.0
mutagen==1.45.1
mutesync==0.0.1
mypy-extensions==0.4.3
nad-receiver==0.3.0
natsort==8.0.2
nclib==1.0.1
ndms2-client==0.1.1
nessclient==0.9.15
netaddr==0.8.0
netdata==1.0.1
netdisco==3.0.0
netifaces==0.11.0
netmap==0.7.0.2
nettigo-air-monitor==1.2.1
neurio==0.3.1
nexia==0.9.13
nextcloudmonitor==1.1.0
niko-home-control==0.2.1
niluclient==0.1.2
noaa-coops==0.1.8
nodeenv==1.6.0
noiseprotocol==0.3.1
notifications-android-tv==0.1.3
notify-events==1.0.4
nsapi==3.0.5
nsw-fuel-api-client==1.1.0
nuheat==0.3.0
numato-gpio==0.10.0
numpy==1.21.4
oasatelematics==0.3
oauth2client==4.1.3
oauthlib==3.1.1
objgraph==3.4.1
oemthermostat==1.1.1
omnilogic==0.4.5
ondilo==0.2.0
onkyo-eiscp==1.2.7
onvif-zeep-async==1.2.0
open-garage==0.2.0
open-meteo==0.0.1
openerz-api==0.1.0
openevsewifi==1.1.0
openhomedevice==2.0.1
opensensemap-api==0.1.5
openwebifpy==3.2.7
openwrt-luci-rpc==1.1.11
openwrt-ubus-rpc==0.0.2
OPi.GPIO==0.5.2
oru==0.1.11
orvibo==1.1.1
ovoenergy==1.1.12
oyaml==1.0
p1monitor==1.0.1
packaging==21.3
paho-mqtt==1.6.1
panacotta==0.1
panasonic-viera==0.3.6
pandas==1.3.5
paramiko==2.9.2
passlib==1.7.4
pbr==5.8.0
pcal9535a==0.7
pdunehd==1.3.2
pencompy==0.0.3
pexpect==4.6.0
phone-modem==0.1.1
pi1wire==0.1.0
pi4ioe5v9xxxx==0.0.2
pifacecommon==4.2.2
pifacedigitalio==3.0.5
piglow==1.2.4
pigpio==1.78
pilight==0.1.1
Pillow==9.0.0
pizzapi==0.0.3
pkce==1.0.3
platformdirs==2.4.1
PlexAPI==4.7.1
plexauth==0.0.6
plexwebsocket==0.0.13
pluggy==1.0.0
plugwise==0.8.5
plumbum==1.7.2
plumlightpad==0.0.11
ply==3.11
pmsensor==0.4
poolsense==0.0.8
pprintpp==0.4.0
praw==7.4.0
prawcore==2.3.0
prayer-times-calculator==0.0.5
pre-commit==2.16.0
prettytable==3.0.0
ProgettiHWSW==0.1.1
proliphix==0.4.1
prometheus-client==0.7.1
proto-plus==1.19.8
protobuf==3.19.3
proxmoxer==1.1.1
psutil==5.8.0
ptyprocess==0.7.0
pubnub==5.5.0
pulsectl==20.2.4
pure-pcapy3==1.0.1
pure-python-adb==0.3.0.dev0
pushbullet.py==0.11.0
pushover-complete==1.1.1
pvo==0.2.0
pwmled==1.6.7
py==1.11.0
py-canary==0.5.1
py-cpuinfo==8.0.0
py-melissa-climate==2.1.4
py-nextbusnext==0.1.5
py-nightscout==1.2.2
py-schluter==0.1.7
py-synologydsm-api==1.0.5
py-vapid==1.8.2
py-zabbix==1.1.7
py17track==2021.12.0
pyads==3.2.2
pyaehw4a1==0.3.9
pyaes==1.6.1
pyaftership==0.1.2
pyairnow==1.1.0
pyairvisual==5.0.9
pyalmond==0.0.2
pyarlo==0.2.4
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyatag==0.3.5.3
pyatmo==6.2.2
pyAtome==0.1.1
pyatv==0.9.8
pybalboa==0.13
pybbox==0.0.5a0
pyblackbird==0.5
pybotvac==0.0.23
pycares==4.1.2
pycarwings2==2.13
pyCEC==0.5.1
pycfdns==1.2.2
pychannels==1.0.0
PyChromecast==10.2.3
pycketcasts==1.0.0
pyclimacell==0.18.2
pycmus==0.1.1
pycocotools==2.0.1
pycognito==2022.1.0
pycomfoconnect==0.4
pyControl4==0.0.6
pycoolmasternet-async==0.1.2
pycountry==22.1.10
pycountry-convert==0.7.2
pycparser==2.21
pycryptodome==3.12.0
pycryptodomex==3.12.0
pycsspeechtts==1.0.4
pydaikin==2.7.0
pydanfossair==0.1.0
pydantic==1.9.0
pydeconz==85
pydelijn==0.6.1
pydexcom==0.2.2
PyDispatcher==2.0.5
pydoods==1.0.2
pydroid-ipcam==0.8
pyebox==1.1.4
pyeconet==0.1.14
pyedimax==0.2.1
pyefergy==0.1.5
pyEight==0.2.0
pyEmby==1.8
pyenvisalink==4.0
pyephember==0.3.1
pyeverlights==0.1.0
pyevilgenius==1.0.0
pyezviz==0.2.0.5
pyfido==2.1.1
pyfireservicerota==0.0.43
pyflic==2.0.3
PyFlick==0.0.2
PyFlume==0.6.5
pyflunearyou==2.0.2
pyfnip==0.2
pyforked-daapd==0.1.11
pyfreedompro==1.1.0
pyfritzhome==0.6.2
PyFronius==0.7.1
pyftdi==0.53.3
pyfttt==0.3
pygatt==4.0.5
Pygments==2.11.2
pygtfs==0.1.6
pygti==0.9.2
pyhaversion==21.11.1
pyheos==0.7.2
pyHik==0.3.0
pyhiveapi==0.4.2
pyhomematic==0.1.77
pyhomeworks==0.0.6
pyhumps==3.5.0
pyialarm==1.9.0
pyicloud==0.7.3
pyinsteon==1.0.14
pyintesishome==1.7.6
pyipma==2.0.5
pyipp==0.11.0
pyiqvia==2021.11.0
pyirishrail==0.0.2
pyiss==1.0.1
pyisy==3.0.1
pyitachip2ir==0.0.7
pyjsparser==2.7.1
PyJWT==2.1.0
pykira==0.1.1
pykmtronic==0.3.0
pykodi==0.2.7
pykrakenapi==0.1.8
pykulersky==0.5.2
pykwb==0.0.8
pylacrosse==0.4
pylast==4.2.1
pylaunches==1.2.1
pylgnetcast==0.3.7
pylibrespot-java==0.1.0
pylint==2.12.2
pylitejet==0.3.0
pylitterbot==2021.12.0
pylutron==0.2.8
pylutron-caseta==0.13.0
pymailgunner==1.4
pymata-express==1.19
pymazda==0.2.2
pymediaroom==0.6.4.1
PyMeeus==0.5.11
pymelcloud==2.5.6
PyMetEireann==2021.8.0
pymeteoclimatic==0.0.6
PyMetno==0.9.0
pymfy==0.11.0
pymitv==1.4.3
pymochad==0.2.0
pymodbus==1.5.2
pymonoprice==0.3
pymsteams==0.1.12
PyMVGLive==1.1.4
pymyq==3.1.4
pymysensors==0.22.1
PyNaCl==1.4.0
pynetgear==0.8.0
pynetio==0.1.9.1
PyNINA==0.1.4
pynuki==1.5.2
pynut2==2.1.2
pynws==1.3.2
pynx584==0.5
pynzbgetapi==0.2.0
pyobihai==1.3.1
pyoctoprintapi==0.1.7
pyombi==0.1.10
pyOpenSSL==21.0.0
pyopenuv==2021.11.0
pyopnsense==0.2.0
pyoppleio==1.0.5
pyotgw==1.1b1
pyotp==2.6.0
pyoverkiz==1.1.1
pyowm==3.2.0
pyownet==0.10.0.post1
pyparsing==2.4.7
pypca==0.0.7
pypck==0.7.12
pypjlink2==1.2.1
pyplaato==0.0.15
pypoint==2.2.1
pyprof2calltree==1.4.5
pyprosegur==0.0.5
pyps4-2ndscreen==1.2.0
Pypubsub==4.0.3
PyQRCode==1.2.1
pyquery==1.4.3
pyqvrpro==0.52
pyqwikswitch==0.93
pyrail==0.0.3
pyrainbird==0.4.3
Pyrebase4==4.5.0
pyrecswitch==1.0.2
pyrepetierng==0.1.0
pyRFC3339==1.1
pyRFXtrx==0.27.0
pyrisco==0.3.1
pyrituals==0.0.6
PyRMVtransport==0.3.2
pyroute2==0.6.5
pyroute2.core==0.6.5
pyroute2.ethtool==0.6.5
pyroute2.ipdb==0.6.5
pyroute2.ipset==0.6.5
pyroute2.ndb==0.6.5
pyroute2.nftables==0.6.5
pyroute2.nslink==0.6.5
pyrsistent==0.18.0
pyruckus==0.12
pysabnzbd==1.1.0
pysaj==0.0.16
pySDCP==1
pysensibo==1.0.3
pyserial==3.5
pyserial-asyncio==0.5
pysesame2==1.0.1
Pysher==1.0.1
pysiaalarm==3.0.2
pysignalclirestapi==0.3.4
pyskyqhub==0.1.4
pysma==0.6.10
pysmappee==0.2.29
pysmartapp==0.3.3
pysmartthings==0.7.6
pysmarty==0.8
pysmb==1.2.7
pysmi==0.3.4
pysml==0.0.5
pysnmp==4.4.12
PySocks==1.7.1
pysoma==0.0.10
pyspcwebgw==0.4.0
pysqueezebox==0.5.5
pysuez==0.1.19
PySyncThru==0.7.10
pytankerkoenig==0.0.6
pytautulli==21.11.0
pytest==6.2.5
pytest-cov==2.12.1
pytest-mock==3.6.1
pytest-rerunfailures==10.2
pytest-runner==5.3.1
pytfiac==0.4
pythinkingcleaner==0.0.3
python-awair==0.2.1
python-blockchain-api==0.0.2
python-clementine-remote==1.0.1
python-dateutil==2.8.2
python-didl-lite==1.3.2
python-digitalocean==1.13.2
python-dotenv==0.19.2
python-ecobee-api==0.2.14
python-engineio==3.14.2
python-etherscan-api==0.0.3
python-family-hub-local==0.0.2
python-forecastio==1.4.0
python-gc100==1.0.3a0
python-gitlab==1.6.0
python-hpilo==4.3
python-http-client==3.3.5
python-izone==1.2.3
python-join-api==0.0.6
python-jose==3.3.0
python-juicenet==1.0.2
python-jwt==3.3.2
python-kasa==0.4.0
python-magic==0.4.24
python-miio==0.5.9.2
python-mpd2==3.0.4
python-mystrom==1.1.2
python-nest==4.1.0
python-opendata-transport==0.3.0
python-openzwave-mqtt==1.4.0
python-picnic-api==1.1.0
python-qbittorrent==0.4.2
python-ripple-api==0.0.3
python-singleton==0.1.2
python-slugify==4.0.1
python-smarttub==0.0.29
python-sochain-api==0.0.2
python-socketio==4.6.1
python-songpal==0.12
python-tado==0.12.0
python-telegram-bot==13.1
python-twitch-client==0.6.0
python-vlc==1.1.2
python-whois==0.7.3
pythonegardia==1.0.40
pyTibber==0.21.7
pytile==2021.10.0
pytouchline==0.7
pytraccar==0.10.0
pytradfri==7.2.1
pytrafikverket==0.1.6.2
PyTransportNSW==0.1.1
PyTurboJPEG==1.6.5
pytz==2021.3
pytz-deprecation-shim==0.1.0.post0
pyudev==0.22.0
pyunifiprotect==1.6.3
pyuptimerobot==21.11.0
pyusb==1.2.1
pyvera==0.3.13
pyversasense==0.0.6
pyvesync==1.4.2
PyViCare==2.15.0
pyvizio==0.1.57
pyvlx==0.2.19
pyvolumio==0.1.3
pyW215==0.7.0
pyW800rf32==0.1
pywebpush==1.9.2
pywemo==0.7.0
pywilight==0.0.70
pywinusb==0.4.2
pyxeoma==1.4.1
PyXiaomiGateway==0.13.4
PyYAML==6.0
pyzbar==0.1.7
pyzerproc==0.4.8
qnapstats==0.4.0
quantum-gateway==0.0.6
RachioPy==1.0.3
radiotherm==2.1.0
raincloudy==0.0.7
raspyrfm-client==1.2.8
ratelimit==2.2.1
readlike==0.1.3
regenmaschine==2021.10.0
regex==2021.11.10
renault-api==0.1.4
ReParser==1.4.3
repoze.lru==0.7
requests==2.27.1
requests-file==1.5.1
requests-futures==1.0.0
requests-oauth==0.4.1
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
responses==0.17.0
RestrictedPython==5.2
rfc3986==1.5.0
rfk101py==0.0.1
rflink==0.0.58
rich==10.16.2
ring-doorbell==0.7.2
ritassist==0.9.2
rjpl==0.3.6
rocketchat-API==0.6.1
rokuecp==0.10.0
roombapy==1.6.5
roonapi==0.0.38
rova==0.2.1
rpi-bad-power==0.1.0
rsa==4.8
RtmAPI==0.7.2
rtsp-to-webrtc==0.4.0
russound==0.1.9
russound-rio==0.1.7
Rx==3.2.0
rxv==0.7.0
s3transfer==0.5.0
samsungctl==0.7.1
samsungtvws==1.6.0
satel-integra==0.3.4
scapy==2.4.5
schedule==1.1.0
schiene==0.23
screenlogicpy==0.5.4
scsgate==0.1.0
semver==2.13.0
sendgrid==6.8.2
sense-energy==0.9.3
sense-hat==2.2.0
sentry-sdk==1.5.2
sepaxml==2.0.0
sgmllib3k==1.0.0
sharkiqpy==0.1.8
sharp-aquos-rc==0.3.2
shodan==1.26.0
simplehound==0.3
simplejson==3.17.6
simplepush==1.1.4
simplisafe-python==2021.11.2
siobrultech-protocols==0.5.0
sisyphus-control==3.1.2
six==1.16.0
skybellpy==0.6.3
slackclient==2.5.0
sleekxmppfs==1.3.4
sleepyq==0.8.1
slixmpp==1.7.1
smart-meter-texas==0.4.7
SmartHab==0.21
smbus-cffi==0.5.1
smbus2==0.4.1
smhi-pkg==1.0.15
sn3218==1.2.7
snapcast==2.1.3
sniffio==1.2.0
snitun==0.30.0
snowballstemmer==2.2.0
soco==0.25.3
solaredge==0.0.2
solaredge-local==0.2.0
solax==0.2.9
somecomfort==0.8.0
somfy-mylink-synergy==1.0.6
sonarr==0.3.0
soupsieve==2.3.1
speak2mary==1.4.0
speedtest-cli==2.1.3
Sphinx==4.3.2
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
spiderpy==1.6.1
spidev==3.5
spotipy==2.19.0
SQLAlchemy==1.4.27
sqlitedict==1.7.0
srpenergy==1.3.2
srptools==1.0.1
sseclient-py==1.7.2
starkbank-ecdsa==2.0.3
starline==0.1.5
starlingbank==3.2
statsd==3.2.1
stdiomask==0.0.6
steamodd==4.21
stevedore==3.5.0
stookalert==0.1.4
streamlabswater==1.0.1
stringcase==1.2.0
subarulink==0.3.12
sucks==0.9.4
sunwatcher==0.2.1
surepy==0.7.2
swisshydrodata==0.1.0
synology-srm==0.2.0
systembridge==1.1.5
tabulate==0.8.9
tailer==0.4.1
tailscale==0.1.6
tank-utility==1.4.0
tapsaff==0.2.1
tellcore-net==0.4
tellcore-py==1.1.2
tellduslive==0.10.11
tellsticknet==0.1.2
temescal==0.3
temperusb==1.5.3
tenacity==8.0.1
tesla-powerwall==0.3.12
tesla-wall-connector==1.0.1
text-unidecode==1.3
thermoworks-smoke==0.1.8
thingspeak==1.0.0
tikteck==0.4
titlecase==2.3
tmb==0.0.4
todoist-python==8.0.0
tololib==0.1.0b3
toml==0.10.2
toonapi==0.2.1
tornado==6.1
total-connect-client==2021.12
tp-connected==0.0.4
tqdm==4.62.3
transitions==0.8.10
transmissionrpc==0.11
TravisPy==0.3.5
ttls==1.4.2
tuya-iot-py-sdk==0.6.6
twentemilieu==0.5.0
twilio==6.32.0
TwitterAPI==2.7.5
typer==0.4.0
typing-inspect==0.7.1
typing_extensions==4.0.1
tzdata==2021.5
tzlocal==4.1
uamqp==1.5.1
uEagle==0.0.2
ujson==4.3.0
unasync==0.5.0
unifiled==0.11
untangle==1.1.1
upb-lib==0.4.12
upcloud-api==2.0.0
update-checker==0.18.0
uplink==0.9.5
uplink-protobuf==0.1.0
uritemplate==3.0.1
url-normalize==1.4.1
urllib3==1.26.8
urwid==2.1.2
uscisstatus==0.1.1
uvcclient==0.11.0
vallox-websocket-api==2.6.0
vehicle==0.3.1
velbus-aio==2021.11.7
venstarcolortouch==0.15
vilfo-api-client==0.3.2
vincenty==0.1.4
virtualenv==20.13.0
vobject==0.9.6.1
vol==0.1.1
volkszaehler==0.2.1
voluptuous==0.12.2
voluptuous-serialize==2.5.0
volvooncall==0.9.1
vsure==1.7.3
vtjp==0.1.14
vultr==0.1.2
wakeonlan==2.0.1
wallbox==0.4.4
waqiasync==1.0.0
warrant-lite==1.0.4
watchdog==2.1.6
waterfurnace==1.1.0
WazeRouteCalculator==0.14
wcwidth==0.2.5
webcolors==1.11.1
webexteamssdk==1.1.1
websocket-client==1.1.0
websockets==8.1
Werkzeug==2.0.2
whirlpool-sixth-sense==0.15.1
wiffi==1.1.0
wirelesstagpy==0.8.1
withings-api==2.3.2
wled==0.11.0
wolf-smartset==0.1.11
wrapt==1.13.3
WSDiscovery==2.0.0
XBee==2.3.2
xbee-helper==0.0.7
xbox-webapi==2.0.11
xboxapi==2.0.1
xknx==0.18.15
XlsxWriter==3.0.2
xmltodict==0.12.0
xs1-api-client==3.0.0
yalesmartalarmclient==0.3.7
yalexs==1.1.18
yarl==1.7.2
yeelight==0.7.8
yeelightsunflower==0.0.10
youless-api==0.16
youtube-dl==2021.12.17
zeep==4.1.0
zengge==0.2
zeroconf==0.38.1
zha-quirks==0.0.65
zhong-hong-hvac==1.0.9
ziggo-mediabox-xl==1.1.0
zigpy==0.42.0
zigpy-deconz==0.14.0
zigpy-xbee==0.14.0
zigpy-zigate==0.7.3
zigpy-znp==0.6.4
zipp==3.7.0
zm-py==0.5.2
zope.interface==5.4.0
zwave-js-server-python==0.33.0

[notatallshaw]

Actually I rebuilt the requirements files with the above freeze information.

Using these and fixing/removing the constraints file you can install using pip 21.3.1. Though you will need to figure out which libraries are too old for your requirements and look at why they have conflicts by updating them and see what the conflicts are.

Here is a diff on the requirements_all.txt:

-PyRMVtransport==0.3.3
+PyRMVtransport==0.3.2
-aioambient==2021.11.0
+aioambient==2.0.0
-aiobotocore==1.2.2
+aiobotocore==2.1.0
-aioridwell==2021.12.2
+aioridwell==2021.10.0
-aiosteamist==0.3.1
+aiosteamist==0.1.0
-boto3==1.16.52
+boto3==1.20.24
-hole==0.7.0
+hole==0.6.0
-# homeassistant.components.mycroft
-mycroftapi==2.0
-open-meteo==0.2.1
+open-meteo==0.0.1
-py17track==2021.12.2
+py17track==2021.12.0
-pyicloud==0.10.2
+pyicloud==0.7.3
-pymodbus==2.5.3
+pymodbus==1.5.2
-pyoverkiz==1.1.0
+pyoverkiz==1.1.1
-pysml==0.0.6
+pysml==0.0.5
-# homeassistant.components.stiebel_eltron
-pystiebeleltron==0.0.1.dev2
-pytile==2021.12.0
+pytile==2021.10.0
-simplisafe-python==2021.12.2
+simplisafe-python==2021.11.2
-systembridge==2.2.3
+systembridge==1.1.5
-vallox-websocket-api==2.8.1
+vallox-websocket-api==2.6.0

And the diff on requirements.txt:

-aiohttp==3.8.1
+aiohttp==3.7.4.post0
-async_timeout==4.0.2
+async_timeout==3.0.1
-pip>=8.0.3,<20.3

[balloob]

Thanks for digging in!

Is there a place where we can find your experimental pip version?

We definitely have some old packages around for integrations that haven't been touched in years. Sometimes integration requirements get disabled when we raise our minimum Python version. Looks like we'll have to do so here too.

It's too bad that our lax handling of dependencies of our integrations is finally catching up on us. It means we'll need to pro-actively take care of it. Ccurrently we do it once an issue is raised that something doesn't work.

The requirements diff doesn't look too bad although I would hate to have to downgrade things (because it means we can't move forward!)

[pradyunsg]

You can blame reach out to packages like discord-py regarding some of the pain here, asking them to remove the relevant problematic pins. You're basically already seeing the problems described in https://iscinumpy.dev/post/bound-version-constraints/#tldr.

If those packages are using poetry or using something like ~= without good reason, point them to the aforementioned blog post.

[notatallshaw]

Is there a place where we can find your experimental pip version?

Yes but it's not stable so I don't guarantee anything about it.

To explain a little bit about what it's doing, a backtracking optimization landed on pip 21.3: #10479 . This optimization is very good at handling the cases where the solution seems "obvious", i.e. A and B don't agree on what version C should be used, so it checks different versions of A and B until they do. However it's not very good at situations where you add a dependency A and it conflicts with multiple preexisting dependencies e.g. X, Y, and Z.

In the experimental version I am testing I use backjumping in an attempt to make those scenarios more solvable. But it still has many limitations and my implementation at the moment is very hacky and I've not verified it other than testing it in complex scenarios and seeing how it performs.

Here's a tag of the version I used to help here: https://github.com/notatallshaw/pip/tree/homeassistant . I think you can install it like so: python -m pip install git+https://github.com/notatallshaw/pip@homeassistant

It also only helps to a certain extent, allowing to make some of your requirements a little bit more flexible. You can't for example remove every version requirement from the hundreds of requirements and expect it to find a solution, I still had to very manually and iteratively loosen up the requirements. You might want to check how other large projects like apache-airflow handle this? I think there are tools like pip compile that are supposed to help? (never had to look in to it myself).

We definitely have some old packages around for integrations that haven't been touched in years. Sometimes integration requirements get disabled when we raise our minimum Python version. Looks like we'll have to do so here too.

What makes it tricky to find the conflicting packages here is that everything is added at once. I wonder if it's possible you can break your integrations down in to logical sub groups and split your requirements in to those groups? Then at least you can test the issues in the groups and then once resolved see if you can install the groups together.

It's too bad that our lax handling of dependencies of our integrations is finally catching up on us. It means we'll need to pro-actively take care of it. Ccurrently we do it once an issue is raised that something doesn't work.

Python is a tricky language to manage such a large number of requirements. I wonder if you'll have to ultimately take some sort of vendoring approach so you can have different dependencies have different versions of their dependencies.

The requirements diff doesn't look too bad although I would hate to have to downgrade things (because it means we can't move forward!)

Yeah most of them look fine, but there's a few like open-meteo where the downgrade looks very challenging :(.

[jensens]

We had the same problem with our CMS Plone which has lots of dependencies. With current pip main branch and this #10574 PR merged these issues are solved. Did you try it already?
I installed pip like so pip install git+https://github.com/pypa/pip.git#egg=pip

[pradyunsg]

This is different, since they have dependency conflicts in the graph, rather than a dense graph post-resolve.