Packages imported with pip/requirements infrastructure cause import issues

[cpatrick]

For example, import google.auth blocks google.storage blocks google.bigquery, etc.

Here's a minimal reproduction: https://github.com/cpatrick/minimal_bazel_python_bug_repro.

Maybe I'm misunderstanding a feature or option I could be using?

[mattmoor]

With zero context, I suspect this is caused by this issue, but can TAL to confirm after lunch.

Thank you for the repro!

[mattmoor]

Interestingly that does not seem like the same issue.

@duggelz the repro prints out sys.path, which seems to include the root of the bigquery whl's repository, so I can't understand why the binary isn't able to load it? Do you have any great insight here?

[cpatrick]

So, my initial guess is that since multiple packages contain a google root package and the approach taken is one directory per package added to sys.path, the import system finds the first one and doesn't search through the rest of sys.path.

A solution could be to combine all of the requirements-installed python packages into a single dist-packages-style folder, but that's a little bit scary...

I'm hoping y'all have better ideas. 😸

[mattmoor]

@cpatrick Yeah, that would be trouble if that's what's going on.

If that's truly what's going on, yes we might be able to work around it in the pip rules, but it still represents a somewhat fundamental problem with how the imports=[] kwarg manifests (I assume it is the usage here that creates these sys.path entries)

Perhaps we need a better solution for declaring a library to be a part of the top-level namespace than imports=["."], and py_binary should handle it better than just stacking sys.path (perhaps it should blend these in the runfiles or something?).

Thinking out loud...

[cpatrick]

Ah, thanks for the pointer and explanation. Yeah, I think that's what happens. the imports=["."] is a bit too much of a hammer.

Blending in runfiles as pip/setuptools does in site-packages feels like the option most consistent with the standard way of doing this. Looking at my virtualenv for the project that I'm bazel-ifying (or from the requirements in my repro-case) that's what you get--a single tree with google/auth, google/cloud/storage, google/cloud/bigquery, etc. all from different packages.

[duggelz]

The google package is complicated. Basically every package like google.cloud.foo, or google.protobuf, or google.tonsofotherstuff, defines its own "google" package, and the first one wins. So every package has to carefully construct its "google" package which does the right thing if it happens to be the "winner", which can easily change based on almost anything. This doesn't always happen. We don't have a central "owns the google package" authority unfortunately.

So, either 1) subpar doesn't support the right pkg_resources and pkgutil calls, which is completely possible, or 2) the "winning" google package happens to be one of the ones that doesn't correctly initialize itself in the maximally friendly way.

I'll look into it further.

[mattmoor]

Here it is just py_binary, not even PAR :-/

[cpatrick]

Hi folks. Any update on this? Let me know if there's anything I can do to help reproduce or even contribute to fixing the issue.

[jpoehnelt]

Any workarounds for this?

[mattmoor]

@damienmg Anyone from the Bazel/Python side we can include here?

[damienmg]

Afraid not, cc @lberki just in case.

…

On Wed, Oct 18, 2017, 4:28 AM Matt Moore ***@***.***> wrote: @damienmg <https://github.com/damienmg> Anyone from the Bazel/Python side we can include here? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADjHf5a27xX46fdkpfkj6RBlFyM88ZOkks5stWI-gaJpZM4Pfqjm> .