Review the usability of pip's command line API for calling from other processes

[pfmoore]

As the officially supported means of using pip from your program is via its command line interface, we should consider whether our command line API is "automation friendly" in this sense.

Specific things we should look at:

1. Allowing callers to suppress configuration set in the external environment (config files, environment variables). The --user flag is a common culprit here, as it doesn't interact well with certain other options like --target.
2. Allowing callers to explicitly pass through certain environment settings (such as proxy) but not others.

At the moment, we have --isolated, but that's a bit of a sledgehammer option, requiring users to replicate pip's option-handling logic if they want to respect some config (for example, network settings) but not others (for example, --user)

Note that this would also help pip itself, in its test suite and in places like build environment creation.

I'm just adding this as a reminder that we should consider this use case. It may be that there's nothing that we actually need to do (except possibly provide some more detailed recommendations on "how to call pip from your application").

[pradyunsg]

#5400 <- sorta kinda relevant.

[cjerdonek]

One thing that I think is important for this is making it so that all log messages go to stderr instead of stdout. Currently, IIRC logger.info() messages for example go to stdout, which can prevent one from consuming a command's structured output with pipes, etc, since the output gets cluttered with other stuff.

In cases where logger.info() messages are part of a command's API (e.g. pip list), we should be using some other function (e.g. a wrapper around print()) which writes to stdout, and then switch the rest of the logger calls to use stderr.

[wimglenn]

More machine-friendly output formats would help greatly. Currently have to scrape the human-readable logging output from subprocess stdout in johnnydep/pipper.py, even for seemingly simple tasks (such as listing the versions available / getting the distribution download links).