Python RuntimeError during sanity check with python3

[rsarm]

When installing easybuild with pip install easybuild using python3 (I tried with python3.5 and python3.6), the built process fails during the sanity checking step with

RuntimeError: dictionary changed size during iteration

that happens in the block line 173 to 177 of easybuild/tools/environment.py. I put the full traceback in attachment because maybe it's a bit too long.

I tried only with a couple of easyconfigs, for instance Julia-1.2.0-linux-x86_64.eb. It can be reproduced with eb --install-latest-eb-release as well.

[Flamefire]

This is a valid bug. Reduced code sample:

def modify_env(old, new, verbose=True):
    oldKeys = old.keys()
    newKeys = new.keys()
    # Snip
    for key in oldKeys:
        if key not in newKeys:
            _log.debug("Key in old environment found that is not in new one: %s (%s)" % (key, old[key]))
            os.unsetenv(key)
            del os.environ[key]

modify_env(os.environ, env, verbose=False)

As you can see: It iterates over os.environ.keys() and calls del os.environ[key]

Solution: Copy the keys: oldKeys = list(old.keys()) and add a test

[boegel]

This clearly is a bug, and we'll fix it for the next EasyBuild release, but I'm puzzled how we have missed this up until now...

@rsarm Can you try using eb --install-latest-eb-release --debug --logtostdout, which should tell you which environment variable was deleted, which then triggers the bug?

[Flamefire]

in Python 3 the keys method was changed to a view, while it was a list/copy in python 2. And System Python is usually still python 2. I guess that's why it went unnoticed so far

[boegel]

No, there's more going on...

This works fine for me, but it's not supposed to because of the del os.environ in modify_env:

python3 -c 'from easybuild.tools.environment import modify_env; import os; modify_env(os.environ, {})'

I don't understand why this doesn't trigger RuntimeError: dictionary changed size during iteration, while a trivial example like this does:

$ python3
Python 3.7.2 (default, Dec 27 2018, 07:35:06)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> d = {1: 1, 2: 2}
>>> for key in d:
...   del d[key]
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: dictionary changed size during iteration

[boegel]

OK, apparently sufficiently recent versions of Python 3.5 and 3.6 won't raise a RuntimeError because a workaround was put in place, see python/cpython#2409 .

That explains why we didn't run into this during testing, it's only an issue in practice when using a Python 3.5 or 3.6 that is not latest.

[boegel]

fixed in #3140