Log debug and informational messages to stderr only

[chrahunt]

What's the problem this feature will solve?

Commands that generate output for the user (pip list, pip freeze) must be concerned with debug output.

Describe the solution you'd like

As mentioned in #6099:

1. all log messages should go to stderr instead of stdout
2. command API related output should use a dedicated interface that outputs to stdout

Any issue that seeks to make the CLI more usable would benefit from this change. This also avoids issues like #6024.

Alternative Solutions

Originally, ad-hoc redirection of logger output to stderr was used to preserve command output. This could be used again, but it would make logging output from commands inconsistent in pip.

Additional Context

• Review the usability of pip's command line API for calling from other processes #6099

[cjerdonek]

I was planning to do this between the 19.2 and 19.3 release but would be okay with someone else doing it. (I had already done it for subprocess logging in PR #6339.)

I was thinking this should be implemented in three PR's as follows:

1. Switch all command API output to use a function called something like write_output() whose initial implementation just delegates to logging.INFO (so this would be a refactoring PR with no behavior change). This PR will let us review where all the API output is happening.

2. Change the implementation of write_output() from using a logging call to writing directly to stdout.

3. Change the logging configuration to use stderr streams for everything instead of stdout for some. This step will involve simplifying our logging config and in particular merging the subprocess logger introduced in PR Fix #1219: Prevent subprocess stdout and stderr from cluttering pip's stdout. #6339 with the main logger, as they would now both be using the same stderr stream.

[pradyunsg]

One of the things we lose with the proposed changes is the timestamping functionality that we added to --log files.

[cjerdonek]

One of the things we lose with the proposed changes is the timestamping functionality that we added to --log files.

We can address this by having the proposed write_output() also log to the log file the things that would be going to stdout. (The log file is already more verbose than the normal log level, IIRC.)

1. Switch all command API output to use a function called something like `write_output()` whose initial implementation just delegates to `logging.INFO` (so this would be a refactoring PR with no behavior change). This PR will let us review where all the API output is happening.

I have started working on this but am unsure how long it will take.
https://github.com/adamRogerson/pip/tree/6758_api_output