Don't import the world when pip starts up

[cjerdonek]

Currently, when pip starts up, it basically imports its entire code base and all vendored libraries and modules it depends on. This issue is for pip only to import what it needs.

It looks like a large part of this can be done pretty simply by changing commands/__init__.py not to import all the command classes:

pip/src/pip/_internal/commands/__init__.py

Lines 27 to 42 in ddfa401

	commands_order = [
	InstallCommand,
	DownloadCommand,
	UninstallCommand,
	FreezeCommand,
	ListCommand,
	ShowCommand,
	CheckCommand,
	ConfigurationCommand,
	SearchCommand,
	WheelCommand,
	HashCommand,
	CompletionCommand,
	DebugCommand,
	HelpCommand,
	] # type: List[Type[Command]]

And then import only the command class it needs after parsing the command name:

pip/src/pip/_internal/__init__.py

Line 76 in ddfa401

command = commands_dict[cmd_name](isolated=("--isolated" in cmd_args))

This will require storing a mapping of command name to command class location and command "summary."

It looks like this would reduce the startup time by ~20% when I tried a quick test.

[cjerdonek]

Later PR's could move the modules needed for pip startup into a startup subpackage (or similar name) to make it easier to see what code pip's startup depends on.

[cjerdonek]

If in addition to the above, the following pkg_resources import is moved from misc.py

pip/src/pip/_internal/utils/misc.py

Line 21 in ddfa401

from pip._vendor import pkg_resources

to a different location (and get_installed_distributions() is imported in autocompletion.py only when needed), it looks like the improvement increases to > 30%. (This comment on issue #4768 remarked that pip._vendor.pkg_resources is a significant contributor to import time.)

[chrahunt]

Would we want to consider solutions like demandimport in the context of this issue? Just adding

import demandimport
demandimport.ignore("number")
demandimport.enable()

this to the top of the pip script doesn't improve startup time, but I didn't yet investigate why.

[pradyunsg]

Related to #4768.

[cjerdonek]

Would we want to consider solutions like demandimport in the context of this issue?

Personally, I think my preference would be to start out with simple, more explicit methods before relying on a third-party module like that. We haven't really tried basic things yet. As before, one of my concerns is that it could be masking / papering over issues that we could be solving more directly, and it would also maybe make the code harder to understand since the code wouldn't be behaving like people are accustomed to.

[cjerdonek]

this to the top of the pip script doesn't improve startup time, but I didn't yet investigate why.

I think it's because demandimport requires a particular style of import to get the benefit, which we don't use. (Specifically, we import functions and classes directly instead of importing only the module and accessing the function or class as an attribute of the module.)

[pradyunsg]

@cjerdonek Can we close this?