Speedup startup time

[pradyunsg]

There's a lot to gain from speeding up pip's startup time.

For one, pip takes around 600ms to just print the completion text, which is laggy. (as mentioned in #4755). Further, faster startup time might help with the test-suite situation too.

[floam]

I did notice one can save about 80ms by using the option to disable the version check - it'd probably make sense not to do that at all for completions by default. Still, a lot remains to be improved.

[dstufft]

I wonder how much of this time is taken up by importing stuff.

[pradyunsg]

$ time .tox/py36/bin/python -c "from pip._internal import main"

A rudimentary test of running the above 5 times gives me an average of 0.377s.

[benoit-pierre]

Importing pip._vendor.pkg_resources is what's taking most of the time on my machine. They are a few changes in setuptools 36.4 and above that will help a little: https://github.com/pypa/setuptools/blob/master/CHANGES.rst#v3640

[pradyunsg]

master is vendoring setuptools 36.4.0 currently...

pip/src/pip/_vendor/vendor.txt

Line 21 in 4092707

setuptools==36.4.0

[benoit-pierre]

Yes, and it's faster than pip 9.0, but there are some further changes in setuptools 36.5 that might help: https://github.com/pypa/setuptools/blob/master/CHANGES.rst#v3650.

[benoit-pierre]

• pip 9.0: python -c 'import pip._vendor.pkg_resources' 0.35s user 0.03s system 100% cpu 0.385 total
• master: python -c "from pip._vendor import pkg_resources" 0.21s user 0.02s system 99% cpu 0.233 total
• master+updated pkg_resources: python -c "from pip._vendor import pkg_resources" 0.19s user 0.00s system 99% cpu 0.192 total

[pradyunsg]

Oh, nice. That means the next round of vendor updates would bring some speedup. :)

I, personally, am waiting on a new distlib release before giving the vendored libraries another round of updates.

[pradyunsg]

I fired up the profiler and ran pip completion --fish. Here's what I got (all percentages in terms of total time):

• initial import of pip._internal.__init__: 85%
  • pip._internal.cmdoptions.__init__: 79%
    • pip._internal.index: 78% (basically all of the above time is spent in this import)
      • This is where it gets interesting
      • pip._vendor.html5lib: 15.8%
      • pip._vendor.requests: 9.0%
      • pip._vendor.distlib: 5.2%
      • pip._vendor.packaging: 7.1%
      • pip._internal.download: 35.4%
        • pip._internal.utils.logging: 30.4%
          • pip._internal.utils.misc: 29.6%
            • pip._vendor.pkg_resources: 28.8%
• pip._internal.__init__.main(): 15%
  • parseopts(): 0.19%
  • command.main(): 14.7%

[pradyunsg]

PS: Need better profiling tools.

[dstufft]

Lazy importing will probably solve some of those.