How to improve information provided to users when pip backtracks

[ei8fdb]

(Taken over from #8714)
@pradyunsg @brainwane @nlhkabu @georgiamoon @pfmoore @uranusjr

What is the issue?

During a pip install, pip needs to work out which version of a package is a good candidate to install. If it tries one version, finds out it isn’t compatible it needs to “go back” (backtrack) and download an older version, trying that, if it is successful it will continue onto the next package, if not it will continue to backtrack until it finds a compatible version, or it will eventually display "resolution impossible".

If pip starts backtracking during dependency resolution, it does not know how long it will backtrack, and how much computation would be needed.

Why is this a bad experience?

pip does not communicate to the user what it is doing, why it is doing it, how long it will take, and what they can do. It can also take a very long time - "why is pip downloading multiple versions of the same package?".

What do we need to do?

We need to communicate to the user: 1) what pip is doing, 2) why it is doing it , and 3) what the user can do, 4) at what verbosity level we should communicate what information at.

What happens right now?

pip does not give the user any information to understand it is backtracking, or why it is happening. pip does not keep track of the number of times it backtracks.

An example of current output:

(backtrack-pyrax) debian@vps-695a9d26:~/backtrack-pyrax$ pip install pyrax==1.9.8 --use-feature=2020-resolver
Collecting pyrax==1.9.8
Downloading pyrax-1.9.8-py2.py3-none-any.whl (346 kB)
|████████████████████████████████| 346 kB 10.4 MB/s
Collecting python-novaclient==2.27.0
Downloading python_novaclient-2.27.0-py2.py3-none-any.whl (312 kB)
|████████████████████████████████| 312 kB 19.2 MB/s
Collecting PrettyTable<0.8,>=0.7
Downloading prettytable-0.7.2.zip (28 kB)
Collecting iso8601>=0.1.9
Downloading iso8601-0.1.13-py3-none-any.whl (9.3 kB)
Collecting python-keystoneclient>=1.6.0
Downloading python_keystoneclient-3.22.0-py2.py3-none-any.whl (397 kB)
|████████████████████████████████| 397 kB 28.2 MB/s
Downloading python_keystoneclient-3.21.0-py2.py3-none-any.whl (395 kB)
|████████████████████████████████| 395 kB 27.0 MB/s
Downloading python_keystoneclient-3.20.0-py2.py3-none-any.whl (394 kB)
|████████████████████████████████| 394 kB 24.4 MB/s
Downloading python_keystoneclient-3.19.1-py2.py3-none-any.whl (394 kB)
|████████████████████████████████| 394 kB 21.3 MB/s
Downloading python_keystoneclient-3.19.0-py2.py3-none-any.whl (394 kB)
|████████████████████████████████| 394 kB 26.2 MB/s
Downloading python_keystoneclient-3.18.0-py2.py3-none-any.whl (393 kB)
|████████████████████████████████| 393 kB 22.1 MB/s
Downloading python_keystoneclient-3.17.0-py2.py3-none-any.whl (382 kB)
|████████████████████████████████| 382 kB 23.8 MB/s
Downloading python_keystoneclient-3.16.0-py2.py3-none-any.whl (376 kB)
|████████████████████████████████| 376 kB 27.5 MB/s
Downloading python_keystoneclient-3.15.1-py2.py3-none-any.whl (385 kB)
|████████████████████████████████| 385 kB 30.4 MB/s
Downloading python_keystoneclient-3.15.0-py2.py3-none-any.whl (378 kB)
|████████████████████████████████| 378 kB 21.4 MB/s
Downloading python_keystoneclient-3.14.0-py2.py3-none-any.whl (372 kB)
|████████████████████████████████| 372 kB 21.1 MB/s
Downloading python_keystoneclient-3.13.1-py2.py3-none-any.whl (381 kB)
|████████████████████████████████| 381 kB 21.8 MB/s
Downloading python_keystoneclient-3.13.0-py2.py3-none-any.whl (374 kB)
|████████████████████████████████| 374 kB 24.7 MB/s
Downloading python_keystoneclient-3.12.0-py2.py3-none-any.whl (374 kB)
|████████████████████████████████| 374 kB 22.1 MB/s
Downloading python_keystoneclient-3.11.0-py2.py3-none-any.whl (372 kB)
|████████████████████████████████| 372 kB 25.9 MB/s
....etc....

What is wrong with this output?

Right now, pip tells the user nothing. If the user was to leave pip “do its thing” 2 outcomes are possible - pip might install all the packages, or it might fail due to not finding compatible packages. This might take a long time.

Do we need to do UX research?

We think it’s common during a pip install package-name command for pip to backtrack 5 or 6 times and more than 13 backtracks is uncommon. We don’t have any specific data.

We have created a poll asking users for a better idea of how often pip backtracks during an install.

What improvements can we make?

One way to improve this is to give pip the ability to keep track of how often it backtracks, and which candidates it backtracks on – using (backtracking(self, candidate) hook on resolvelib.

Using this, pip could then:

• record the backtracking choices
• count the number of backtracking choices

Proposed solution

We propose pip tells the user 1) why it's is backtracking, and 2) what they need to/can do.

We propose pip:
After the 1st backtrack pip prints: <message 1>
After the 8th backtrack pip prints: <message 1>
After the 13th backtrack pip prints: <message 1><message 2>
Both messages are displayed at verbosity level 0.

Message 1 text

INFO: pip is looking at multiple versions of this package to determine which version is compatible with other requirements. This could take a while.

Message 2 text

INFO: This is a complicated process. You might need to provide the
dependency resolver with stricter constraints to reduce runtime.
If you want to abort this run, you can press Ctrl + C to do so.
    To improve how pip performs, tell us why this happened here: https://github.com/pypa/pip/issues/XXXX

How it would look in a real output:

$ pip install pyrax==1.9.8
Collecting pyrax==1.9.8
Downloading pyrax-1.9.8-py2.py3-none-any.whl (346 kB)
|████████████████████████████████| 346 kB 10.4 MB/s
Collecting python-novaclient==2.27.0
Downloading python_novaclient-2.27.0-py2.py3-none-any.whl (312 kB)
|████████████████████████████████| 312 kB 19.2 MB/s
Collecting PrettyTable<0.8,>=0.7
Downloading prettytable-0.7.2.zip (28 kB)
Collecting iso8601>=0.1.9
Downloading iso8601-0.1.13-py3-none-any.whl (9.3 kB)
Collecting python-keystoneclient>=1.6.0
Downloading python_keystoneclient-3.22.0-py2.py3-none-any.whl (397 kB)
|████████████████████████████████| 397 kB 28.2 MB/s
INFO: pip is looking at multiple versions of this package to determine
which version is compatible with other requirements.
This could take a while.
Downloading python_keystoneclient-3.21.0-py2.py3-none-any.whl (395 kB)
|████████████████████████████████| 395 kB 27.0 MB/s
Downloading python_keystoneclient-3.20.0-py2.py3-none-any.whl (394 kB)
|████████████████████████████████| 394 kB 24.4 MB/s
Downloading python_keystoneclient-3.19.1-py2.py3-none-any.whl (394 kB)
|████████████████████████████████| 394 kB 21.3 MB/s
Downloading python_keystoneclient-3.19.0-py2.py3-none-any.whl (394 kB)
|████████████████████████████████| 394 kB 26.2 MB/s
Downloading python_keystoneclient-3.18.0-py2.py3-none-any.whl (393 kB)
|████████████████████████████████| 393 kB 22.1 MB/s
Downloading python_keystoneclient-3.17.0-py2.py3-none-any.whl (382 kB)
|████████████████████████████████| 382 kB 23.8 MB/s
Downloading python_keystoneclient-3.16.0-py2.py3-none-any.whl (376 kB)
|████████████████████████████████| 376 kB 27.5 MB/s
Downloading python_keystoneclient-3.15.1-py2.py3-none-any.whl (385 kB)
|████████████████████████████████| 385 kB 30.4 MB/s
INFO: pip is looking at multiple versions of this package to determine
which version is compatible with other requirements.
This could take a while.
Downloading python_keystoneclient-3.15.0-py2.py3-none-any.whl (378 kB)
|████████████████████████████████| 378 kB 21.4 MB/s
Downloading python_keystoneclient-3.14.0-py2.py3-none-any.whl (372 kB)
|████████████████████████████████| 372 kB 21.1 MB/s
Downloading python_keystoneclient-3.13.1-py2.py3-none-any.whl (381 kB)
|████████████████████████████████| 381 kB 21.8 MB/s
Downloading python_keystoneclient-3.13.0-py2.py3-none-any.whl (374 kB)
|████████████████████████████████| 374 kB 24.7 MB/s
Downloading python_keystoneclient-3.12.0-py2.py3-none-any.whl (374 kB)
|████████████████████████████████| 374 kB 22.1 MB/s
INFO: pip is looking at multiple versions of this package to determine
which version is compatible with other requirements.
This could take a while.
INFO: This is a complicated process. You might need to provide the
dependency resolver with stricter constraints to reduce runtime.
If you want to abort this run, you can press Ctrl + C to do so.
To improve how pip performs, tell us why this happened here: https://github.com/pypa/pip/issues/XXXX
Downloading python_keystoneclient-3.11.0-py2.py3-none-any.whl (372 kB)
|████████████████████████████████| 372 kB 25.9 MB/s

Documentation

We'll also add documentation to the user guide giving users an explanation of what backtracking is, why is happens, and possible ways to reduce it from occuring. Covered by #9039.

Better solution - using time

A better solution would be to tell the user how long this backtrack will take, and show progress.

There is no way to distingush between small number of steps on “big package downloaded twice” vs lots of steps of “small package downloaded manyyyy times”. And showing time progress might be very difficult to do.

Design rationale

The information here explains the UX design decisions we've made.

This is based on the “progressive disclosure” approach. It is a common UI design pattern. pip will give the user more information the longer the backtracking goes on, because it is more likely the user may want/need to abort the process.

Message 1 - provides the user with the right amount of information at the right time - after the first number of backtracks, pip tells the user why this is happening and it will take time.

Message 2 - More than likely, pip will not need to backtrack any more. If it does, it then provides the user with more information - this time about what they can do if they do not want it to continue.

If we include a link to a GH issue, users can provide information about the situations where backtracking exceeded 13 tries, allowing us to get a better idea of the necessary trigger points, and possibly improve the situation.

Related GH issues

#8713
#8683
#8922
#8883

How you can help

1. Message trigger points

If you'd like to help us improve the trigger points for message 1 and 2, you can complete this short poll (1 question!).

2. Message text

Are these messages acceptable? If you have suggestions, please give some explanation why.

3. Github issue link

We propose opening a new Github issue to track situations where message 2 is displayed. Using a Github issue template we can ask users specific questions.

Any other suggestions?

[pfmoore]

Nice explanation of the issue! 🙂

One thought that occurred to me is that we should also try to ensure that the "progressive" messages don't get lost in among other output - your "how it would look in real output" example has this problem. Maybe coloured output would help here, or careful use of indentation. The broader question of "how do we highlight the important parts of pip's output" is probably a separate issue, though.

[pradyunsg]

The broader question of "how do we highlight the important parts of pip's output" is probably a separate issue, though.

psst #4649.

[ei8fdb]

One thought that occurred to me is that we should also try to ensure that the "progressive" messages don't get lost in among other output - your "how it would look in real output" example has this problem.

Yes you're right - this is a possibility, mainly due to 1) the scrolling nature of pip output and 2) the amount of text outputted by pip at verbosity 0. In hierarchy of what's important to know, this output it is informational - "you might like to know that I have to try older versions".

Maybe coloured output would help here, or careful use of indentation.

Indentation might help - do you have suggestions on how to indent the message? That'd be helpful to see.

I don't see the use of colour being necessary here - the backtracking is not an error. It's very possible pip will successfully install the user's chosen package.

The way to deal with this is a redesign of the pip output - to provide the user with the right level of information about the most important pip tasks at ech verbosity level - as @pradyunsg mentions.

[pfmoore]

Indentation might help - do you have suggestions on how to indent the message? That'd be helpful to see.

No, I don't, unfortunately. It's something I've thought for some time we could do better, but I really have no good ideas. I was sort of hoping that UX expertise would mean you'd magically know all the answers 🙂

Joking aside, I feel that while this is informational content, it's more important than a lot of what pip outputs. In my view, pip is a bit too verbose in its default output. But that feeds into the wider redesign that you mention.

[ei8fdb]

No, I don't, unfortunately. It's something I've thought for some time we could do better, but I really have no good ideas. I was sort of hoping that UX expertise would mean you'd magically know all the answers 🙂

No magic answers. @pradyunsg's suggestion of the message being printed continuously after initial 8 backtracks was to keep it visible to the user.

Joking aside, I feel that while this is informational content, it's more important than a lot of what pip outputs.

To flip that around - a lot of what pip outputs is not important. Again, the answer here is to redesign pip's output.

So far 6/7 respondents to the poll have said they've never noticed pip backtracking, or it has never.

This was put together as an initial design. Based on user feedback (to the GH issue) if we find out more we can iterate this.

In my view, pip is a bit too verbose in its default output. But that feeds into the wider redesign that you mention.

I don't disagree with you on that! During pip install juypter, at verbosity 0 pip outputs 150 lines, while at verbosity 1 it outputs ~ 6950 lines. If it's OK, can we have that discussion in a different thread?

[pradyunsg]

So far 6/7 respondents to the poll have said they've never noticed pip backtracking, or it has never.

As was flagged elsewhere, this might be due to some respondents never seeing that behavior on the old resolver.

[pradyunsg]

If it's OK, can we have that discussion in a different thread?

Yes -- please feel free to use #4649 for this. :)

[ei8fdb]

@uranusjr made me aware that the current pip resolver does not do backtracking. I wasn't aware of this!
Therefore asking pip users about (non-existent) backtracking is not going to get us any data! Hence I'll retire the survey, and suggest we keep with the trigger points.