Platform detection is wrong on x32

[mithrandi]

pip/src/pip/_internal/pep425tags.py

Lines 135 to 138 in f038fe4

	if result == "linux_x86_64" and _is_running_32bit():
	# 32 bit Python program (running on a 64 bit Linux): pip should only
	# install and run 32 bit compiled extensions in that case.
	result = "linux_i686"

This code assumes that i686 is the only way to have 32-bit pointers on x86_64, but the x32 ABI is another way for this to happen, and i686 binaries won't work on x32.

[pradyunsg]

I have little clue about what happens here. Maybe some other @pypa/pip-committers knows.

[pfmoore]

Not me...

[lamby]

One way to reproduce this would be to use i386 Debian with the amd64 kernel. Let me know if you'd like anything specifically run in this environment.

[pfmoore]

Can you demonstrate a specific pip command line invocation that does the wrong thing on that platform? I imagine that would help someone with the relevant understanding of the code to debug the issue.

[pradyunsg]

Pinging @dholth and @ncoghlan (from the PEP).

[ncoghlan]

If there's a Linux platform mode other than i686 and x86_64, CPython core dev doesn't know anything about it either.

[mithrandi]

One way to reproduce this would be to use i386 Debian with the amd64 kernel. Let me know if you'd like anything specifically run in this environment.

I think you mean x32 Debian. i386 Debian does the "right" thing here.

[mithrandi]

You can see the problem in action here on a Debian system (needs a kernel booted with syscall.x32=y):

$ sudo apt-get install sbuild schroot
…
$ sudo sbuild-createchroot --arch=x32 --keyring=/usr/share/keyrings/debian-ports-archive-keyring.gpg --include=debian-ports-archive-keyring unstable /srv/chroot/sid-x32 http://cdn-fastly.deb.
debian.org/debian-ports
…
$ sudo schroot -c unstable-x32-sbuild -u root --directory /root
# apt-get update && apt-get install virtualenv python python-dev
…
# virtualenv venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in /root/venv/bin/python2
Also creating executable in /root/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
# venv/bin/pip install lxml
Collecting lxml
  Downloading https://files.pythonhosted.org/packages/36/a6/e2692daf930575321a574a327f1bdaa45e1b1e0d68ad051ad2e3fd8e48eb/lxml-4.2.1-cp27-cp27mu-manylinux1_i686.whl (5.2MB)
    100% |################################| 5.3MB 8.7MB/s
Installing collected packages: lxml
Successfully installed lxml-4.2.1
# venv/bin/python -c 'import lxml.etree'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /root/venv/local/lib/python2.7/site-packages/lxml/etree.so: cannot open shared object file: No such file or directory

The import fails because pip downloaded and installed an i686 wheel, so the shared object is of the wrong arch/ABI:

# file /root/venv/local/lib/python2.7/site-packages/lxml/etree.so
/root/venv/local/lib/python2.7/site-packages/lxml/etree.so: ELF 32-bit LSB pie executable Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=187c5f1570549508a33153b9636a8141e92bc56d, with debug_info, not stripped
# ldd /root/venv/local/lib/python2.7/site-packages/lxml/etree.so
        not a dynamic executable

If you force a source installation, then it works:

# apt-get install libxml2-dev libxslt1-dev && virtualenv venv2 && venv2/bin/pip install --no-binary :all: lxml
…
# file /root/venv2/local/lib/python2.7/site-packages/lxml/etree.so
/root/venv2/local/lib/python2.7/site-packages/lxml/etree.so: ELF 32-bit LSB pie executable x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=365058be218a3855bb70df187322a6e05370da4f, with debug_info, not stripped
# ldd /root/venv2/local/lib/python2.7/site-packages/lxml/etree.so
        linux-vdso.so.1 (0xffb7f000)
        libxslt.so.1 => /usr/lib/x86_64-linux-gnux32/libxslt.so.1 (0xf7722000)
…

But note that pip would have built and cached a wheel tagged "i686" that contains x32 shared objects if binaries weren't disabled, so this wheel would not work on an actual i686 system.

[ehashman]

I was able to reproduce this bug and I think I have a working patch. Let me attempt to explain what's going on here.

pip/src/pip/_internal/pep425tags.py

Lines 134 to 138 in f038fe4

	result = distutils.util.get_platform().replace('.', '_').replace('-', '_')
	if result == "linux_x86_64" and _is_running_32bit():
	# 32 bit Python program (running on a 64 bit Linux): pip should only
	# install and run 32 bit compiled extensions in that case.
	result = "linux_i686"

In L134, we use distutils.utils.get_platform() to detect the current platform. For a Linux system, this is equivalent to getting the running kernel's architecture from uname. So if we are running Linux on a 64-bit machine, it will return "linux_x86_64" and so on.

There are three Linux ABIs that I am aware of: "linux_x86_64" (64-bit), "linux_i686" (32-bit), "linux_x32" (another flavour of 32-bit ABI). Lines 135-138 are intended to handle the unusual but possible case in which you have a 64-bit kernel but are running 32-bit binaries in userspace.

The code assumes that if this happens, the user must be using the 32-bit "linux_i686" ABI. However, as @mithrandi reports, this assumption is incorrect. We could have compiled userspace against the "linux_x32" ABI instead. When that's the case, we get a platform mismatch of "linux_i686" on a machine with the "linux_x32" ABI. Hence, when we download a dependency that doesn't have pure wheels, pip will incorrectly fetch i686 wheels where they're available: note that lxml-4.2.1-cp27-cp27mu-manylinux1_**i686**.whl was retrieved in the session output above.

Since we're using the x32 ABI, the i686 ELF interpreter isn't available, so when we try to run the wheels, the system barfs. This is why we get an ImportError: /root/venv/local/lib/python2.7/site-packages/lxml/etree.so: cannot open shared object file: No such file or directory error---pip can't find the right ELF interpreter.

I determined that platform.machine() returns "x86_64" on the "linux_x86_64" kernel/"linux_x32" userspace combo, and "i686" on the "linux_x86_64" kernel/"linux_i686" userspace combo. Hence, we can use this to differentiate between the two, and that's the basis of my patch.

[ehashman]

Patch seems to work! Here's the output on the x32 schroot:

(venv) (unstable-x32-sbuild)root@schroot-funtimes:~# pip install -U pip-18.0.dev0.tar.gz   # upgrade to my patched pip in a venv
Processing ./pip-18.0.dev0.tar.gz
  Installing build dependencies ... done
Building wheels for collected packages: pip
  Running setup.py bdist_wheel for pip ... done
  Stored in directory: /root/.cache/pip/wheels/c5/57/c9/a53d3e8f7e467f46e2ceb1c8a634ba2bcdb50bd6510e2fa01b
Successfully built pip
Installing collected packages: pip
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
Successfully installed pip-18.0.dev0
(venv) (unstable-x32-sbuild)root@schroot-funtimes:~# pip install lxml
Collecting lxml
  Downloading https://files.pythonhosted.org/packages/e8/5d/98f56e274bdf17f2e0d9016d1788ca80d26d8987dcd5e1d9416d86ee0625/lxml-4.2.1.tar.gz (4.3MB)
    100% |################################| 4.3MB 5.0MB/s 
Building wheels for collected packages: lxml
  Running setup.py bdist_wheel for lxml ... error
  Complete output from command /root/venv/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-V3gkiM/lxml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-yVIV6w --python-tag cp27:
  Building lxml version 4.2.1.
  Building without Cython.
  ERROR: /bin/sh: 1: xslt-config: not found
  
  ** make sure the development packages of libxml2 and libxslt are installed **
  
...
    Compile failed: command 'x86_64-linux-gnux32-gcc' failed with exit status 1
    cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitdTl16m.c -o tmp/xmlXPathInitdTl16m.o
    /tmp/xmlXPathInitdTl16m.c:1:10: fatal error: libxml/xpath.h: No such file or directory
     #include "libxml/xpath.h"
              ^~~~~~~~~~~~~~~~
    compilation terminated.
    *********************************************************************************
    Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
    *********************************************************************************
    error: command 'x86_64-linux-gnux32-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/root/venv/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-V3gkiM/lxml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-HWDUFB/install-record.txt --single-version-externally-managed --compile --install-headers /root/venv/include/site/python2.7/lxml" failed with error code 1 in /tmp/pip-install-V3gkiM/lxml/

This is great! We got the pure wheel and compilation failed because I lack the dev headers :)

On i686:

(venv) (unstable-i386-sbuild)root@schroot-funtimes:~# pip install -U pip-18.0.dev0.tar.gz 
Processing ./pip-18.0.dev0.tar.gz
  Installing build dependencies ... done
Building wheels for collected packages: pip
  Running setup.py bdist_wheel for pip ... done
  Stored in directory: /root/.cache/pip/wheels/c5/57/c9/a53d3e8f7e467f46e2ceb1c8a634ba2bcdb50bd6510e2fa01b
Successfully built pip
Installing collected packages: pip
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
Successfully installed pip-18.0.dev0
(venv) (unstable-i386-sbuild)root@schroot-funtimes:~# pip install lxml
Collecting lxml
  Downloading https://files.pythonhosted.org/packages/36/a6/e2692daf930575321a574a327f1bdaa45e1b1e0d68ad051ad2e3fd8e48eb/lxml-4.2.1-cp27-cp27mu-manylinux1_i686.whl (5.2MB)
    100% |################################| 5.3MB 3.9MB/s 
Installing collected packages: lxml
Successfully installed lxml-4.2.1

We still get the correct wheels! So I'm pretty convinced this worked, and it shouldn't break anything else as the only logic affected is inside that if case.

[lamby]

I think you mean x32 Debian. i386 Debian does the "right" thing here.

Ah, whoops, ignore me :)