Build Logic Refactor

[pradyunsg]

This is an issue for the broader design discussions for the overall build logic refactor work that I'll be doing.

Related: #5051
Tracking project board: https://github.com/pypa/pip/projects/3

---

@pypa/pip-committers I was thinking about this a bit and IMO we should try to keep the "broad" discussion here since it'd probably make life easier for me to be able to look back and reference things. I've also created a project board to better organize myself while working on this.

[pradyunsg]

Just filed #6622 containing the next step - moving logic into SourceDistribution.

[cjerdonek]

What is the relationship of this issue to the resolver issue (#6536)?

[pradyunsg]

This cleans up things so that adapting pip to resolvelib's abstractions becomes easier.

Plus, I'll be less scared about having broken the general build process while swapping out the resolver logic. 🙃

[chrahunt]

It's not really a design consideration, but the refactoring may be easier if the TODOs and FIXMEs in e.g. InstallationRequirement were made into individual issues so that could be addressed in turn.

[pradyunsg]

@cjerdonek I notice that my previous response is super terse, so I'll elaborate a bit.

I think this project, is closely related to the actually implementing the resolver.

We have to perform the fetch-build-query during our resolution process. Ideally, the resolver should not see that this is happening (i.e. operate on a more "abstract" space, like the resolvelib/zazo abstractions). This, however, also means that we need to make it easier to actually modify the resolution and build logic, which have been entangled in the codebase since I started working on pip. 🙃

In essence, what this project is aimed at doing is make the build logic code easier to reason about and follow, which in turn would make it easier to separate it from our resolution logic and make life easier when we do go ahead with the new resolution logic -- #6536 is for figuring out the communication and overall approach for that.

Another thing I'll note is that, this was the main blocker/issue/tricky-thing-that-I-sunk-time-into when I was working on a weekend-prototype (https://github.com/pradyunsg/pip/tree/resolver/make-it-inside). The build logic essentially had to be moved around a fair bit, and I realized that it'll be a worthwhile investment to decouple the two, and refactoring/improving the build logic was a great way to do that. :)

[pradyunsg]

@chrahunt and I had a couple of voice calls over the past few days, where we discussed this (pip's build logic). We discussed how the builds "flow" and what information we have available at each stage.

To that end, @chrahunt has a GitHub Gist containing both a mental model that we came up with to understand how the build logic works currently as well as a structured file, for potentially generating insights from. All this is a little roughly sketched out right now, and we're still figuring things out in greater detail.

Inputs are definitely welcome. :)

[pradyunsg]

I had an itch to scratch w/ graphviz and this problem lends itself perfectly to being visualized with a directed graph. @dreampuf helpfully built GraphvizOnline, which I used to prototype a graph that shows the code flow that we'd want to follow.

There are definitely a lot more ways to visualize this but my goal here was to have something I can look at later and make sense of, which this is. :)

Link to an online preview of the graph I came up with, since the actual image is too big to reasonably embed here.

Notes:

• double border => artifact that the user can provide as inputs.
• use the dot engine (it's the best one for directed graphs).
• use svg format, to be able to view the entire image.
• it's only visualizing the flow of data, not the creation/state of metadata or available information.

[pradyunsg]

@techalchemy and I had a call recently, and we discussed some items related to this and related to the resolver situation. The following are (some of?) the actionable notes I have from our conversation:

• The most complicated part of this refactor will be directory handling within InstallRequirement. (we both, just, strongly agreed)
• Useful cleaner abstractions:
  • ModernMetadataGenerator(), LegacyMetadataGenerator() for the logic behind "prepare metadata" operations
  • WheelInstaller(), LegacyEditableInstaller(), LegacyInstaller() for the install mechanisms
• Should rename InstallationCandidate to something that does not conflict with the Candidate from resolvelib's model (current: IndexCandidateInfo)

Broader notes related to the resolver:

• "candidate, as state machine that only mutates on transitions" model makes sense -- @chrahunt and I came up with in our prior discussion where we modeled the build flows.
• Should build a better abstraction model for RequirementSet (basically, don’t manipulate in-place and create a new “CandidateSet” for tracking chosen candidates)
• In resolvelib model, we'd let ‘Provider’ populate & prepare requirements for installation, installation order, etc.
  • Current resolver's interface is the basically the same as resolvelib, except it modifies the RequirementSet in-place. (see point above)