`thread` module is missing

[glyph]

This breaks other parts of the standard library.

It broke at some point in the last week.

$ docker run --rm -it pypy:3
Python 3.6.12 (db1e853f94de, Nov 18 2020, 09:49:19)
[PyPy 7.3.3 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> import crypt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/pypy/lib-python/3/crypt.py", line 3, in <module>
    import _crypt
  File "/opt/pypy/lib_pypy/_crypt/__init__.py", line 7, in <module>
    import thread
ModuleNotFoundError: No module named 'thread'

[tianon]

Looks like this was broken in the 7.3.3 release. 😞

Trying to figure out what happened and how far this goes -- so far, both the Python 3.6 and 3.7 branches are definitely affected, but 2.7 isn't (which isn't really unexpected).

(Not seeing anything obviously related already reported to https://foss.heptapod.net/pypy/pypy/-/issues.)

[tianon]

Confirmed that pypy:3-7.3.2 works fine, so this is definitely something related to 7.3.3.

Not seeing anything super obvious in the released tarballs from https://downloads.python.org/pypy/:

Diff:

$ diff -u <(tar -tf pypy3.6-v7.3.2-linux64.tar.bz2 | cut -d/ -f2-) <(tar -tf pypy3.6-v7.3.3-linux64.tar.bz2 | cut -d/ -f2-)
--- /dev/fd/63	2020-11-25 17:00:18.805503793 -0800
+++ /dev/fd/62	2020-11-25 17:00:18.805503793 -0800
@@ -408,7 +408,6 @@
 lib_pypy/_pwdgrp_build.py
 lib_pypy/_marshal.py
 lib_pypy/_lzma_build.py
-lib_pypy/_decimal_cffi.pypy36-pp73-x86_64-linux-gnu.so
 lib_pypy/__init__.py
 lib_pypy/_decimal_build.py
 lib_pypy/_audioop_cffi.pypy36-pp73-x86_64-linux-gnu.so
@@ -439,6 +438,7 @@
 lib_pypy/_sha3/kcp/KeccakP-1600-SnP-opt32.h
 lib_pypy/_sha3/__pycache__/
 lib_pypy/_sha3/_sha3_cffi.pypy36-pp73-x86_64-linux-gnu.so
+lib_pypy/__decimal.py
 lib_pypy/readline.py
 lib_pypy/marshal.py
 lib_pypy/_pypy_winbase_build.py
@@ -783,6 +783,8 @@
 lib_pypy/_curses_build.py
 lib_pypy/_crypt/
 lib_pypy/_crypt/__init__.py
+lib_pypy/hpy/
+lib_pypy/hpy/universal.py
 lib_pypy/grp.py
 lib_pypy/_pypy_testcapi.py
 lib_pypy/_blake2/
@@ -828,7 +830,6 @@
 lib_pypy/_sha256.py
 lib_pypy/resource.py
 lib_pypy/_curses_panel.py
-lib_pypy/_decimal.py
 lib_pypy/_ctypes/
 lib_pypy/_ctypes/__init__.py
 lib_pypy/_ctypes/builtin.py
@@ -1278,7 +1279,6 @@
 lib-python/3/encodings/cp737.py
 lib-python/3/encodings/euc_jis_2004.py
 lib-python/3/encodings/__pycache__/
-lib-python/3/encodings/__pycache__/ascii.pypy36.pyc
 lib-python/3/encodings/__pycache__/aliases.pypy36.pyc
 lib-python/3/encodings/__pycache__/latin_1.pypy36.pyc
 lib-python/3/encodings/__pycache__/__init__.pypy36.pyc

[tianon]

It seems like https://foss.heptapod.net/pypy/pypy/-/commit/36b29935f3af98f071873218fb3d15200a54bc80 might be the culprit (adding import thread to crypt where it didn't exist previously, and that commit doesn't appear to exist until 7.3.3.rc1)

[glyph]

Reported upstream here: https://foss.heptapod.net/pypy/pypy/-/issues/3395

[mattip]

This was fixed on PyPy HEAD and will be part of the next release. It can be patched in the pure-python file lib_pypy/_crypt/__init__.py (there was another bug fixed in https://foss.heptapod.net/pypy/pypy/-/issues/3378):

diff -r 40928263b3ca -r 7414ff6076ed lib_pypy/_crypt/__init__.py
--- a/lib_pypy/_crypt/__init__.py       Sun Oct 11 18:23:06 2020 +0300
+++ b/lib_pypy/_crypt/__init__.py       Sat Feb 06 20:53:19 2021 +0200
@@ -4,8 +4,8 @@
 
 import sys
 import cffi
-import thread
-_lock = thread.allocate_lock()
+import _thread
+_lock = _thread.allocate_lock()
 
 try: from __pypy__ import builtinify
 except ImportError: builtinify = lambda f: f
@@ -22,12 +22,20 @@
 
 @builtinify
 def crypt(word, salt):
+    # Both arguments must be str on CPython, but are interpreted as
+    # utf-8 bytes.  The result is also a str.  For backward
+    # compatibility with previous versions of the logic here
+    # we also accept directly bytes (and then return bytes).
     with _lock:
-        if isinstance(word, str):
-            word = word.encode('ascii')
+        arg_is_str = isinstance(word, str)
+        if arg_is_str:
+            word = word.encode('utf-8')
         if isinstance(salt, str):
-            salt = salt.encode('ascii')
+            salt = salt.encode('utf-8')
         res = lib.crypt(word, salt)
         if not res:
             return None
-        return ffi.string(res)
+        res = ffi.string(res)
+        if arg_is_str:
+            res = res.decode('utf-8')
+        return res

[tianon]

Thanks, I'll see about applying that -- that's a combination of https://foss.heptapod.net/pypy/pypy/-/commit/16faa2be85839e6ab4fb8ee09298a4d934aab81f and https://foss.heptapod.net/pypy/pypy/-/commit/c63da169246ed972fe90e1c289fc2378236fa852 then, correct?

Separately, do you expect a new release soonish? (We don't do much "packaging" beyond downloading the releases from https://downloads.python.org/pypy/ which is why this affects us and likely affects others outside Docker -- just recompiling some of the shared libraries where necessary. 😅)

[tianon]

Also, it seems clear to me that https://foss.heptapod.net/pypy/pypy/-/commit/16faa2be85839e6ab4fb8ee09298a4d934aab81f applies to 3.6 and 3.7 (but not 2.7), but https://foss.heptapod.net/pypy/pypy/-/commit/c63da169246ed972fe90e1c289fc2378236fa852 seems like it might apply on 2.7 as well; is that accurate?

[mattip]

We will probably do a 3.7/2.7 release in March. There is a slight chance it may include a first alpha of 3.8, but I wouldn't rush to put out a docker image that includes it.

Someone should confirm that the issue occurs on 2.7, and file an issue if needed. I am not sure what the correct typing/encoding of the arguments to crypt.crypt() is on 2.7.

We are not planning another 3.6 release, do you see many people using the 3.6 docker image? CPython 3.6 is in security-fix-only mode, and it is difficult for us to maintain more than one python3 version.

[tianon]

We will probably do a 3.7/2.7 release in March. There is a slight chance it may include a first alpha of 3.8, but I wouldn't rush to put out a docker image that includes it.

For things like this we try to include a (non-default) variant for folks to try it out as soon as we can (the intent being that for a lot of users, docker run pypy:3.8 is going to be easier than the alternatives to try it out and hopefully send feedback / help find bugs 😅).

Someone should confirm that the issue occurs on 2.7, and file an issue if needed. I am not sure what the correct typing/encoding of the arguments to crypt.crypt() is on 2.7.

Ok, fair enough; thanks for the insight - I'll see about getting it applied only on 3.x. 👍

We are not planning another 3.6 release, do you see many people using the 3.6 docker image? CPython 3.6 is in security-fix-only mode, and it is difficult for us to maintain more than one python3 version.

Unfortunately, we don't get much in the way of per-tag stats, but the way the images are tagged all the 3 (and other similarly non-specific) aliases are currently pointing at 3.6 because it was our understanding (from https://morepypy.blogspot.com/2020/11/pypy-733-triple-release-python-37-36.html) that 3.7 is still "beta" so we didn't think it should be the default yet.

Totally understandable on the support levels; I can only imagine the level of effort for supporting any of this often borders on herculean. 😬 ❤️

[mattip]

hmm. As long as we are calling it "beta" that makes sense. I will definitely drop the beta tag on the next release. Thanks for packaging this.