Invalid cross-device link and Invalid argument errors when copying `reprod` directory

[casparvl]

I've faced two issues which I think are related, although I cannot be 100% sure.

Initial issue: invalid cross-device link
When installing any package, I would get a packaging error due to the copy of the reprod dir from the tempdir to the installation directory. Note that the tempdir was on /tmp and the builddir was on another (network) filesystem (/panfs).

== packaging...
ERROR: Traceback (most recent call last):
  File "/usr/lib64/python3.6/shutil.py", line 550, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/eb-jy4z76eo/reprod_20220425030354_267603' -> '/panfs/users/Xcvleeu/icelake/software/EasyBuild/4.5.4/easybuild/reprod'

During handling of the above exception, another exception occurred:

long, other exception...

Traceback (most recent call last):
  File "/panfs/users/achaibi/surf_shared/icelake/software/EasyBuild/4.5.4/lib/python3.6/site-packages/easybuild/tools/filetools.py", line 2707, in move_file
    shutil.move(path, target_path)
  File "/usr/lib64/python3.6/shutil.py", line 561, in move
    symlinks=True)
  File "/usr/lib64/python3.6/shutil.py", line 365, in copytree
    raise Error(errors)
shutil.Error: [('/tmp/eb-jy4z76eo/reprod_20220425030354_267603/easyblocks/pythonpackage.py', '/panfs/users/Xcvleeu/icelake/software/EasyBuild/4.5.4/easybuild/reprod/easyblocks/pythonpackage.py', "[Errno 22] Invalid argument: '/panfs/users/Xcvleeu/icelake/software/EasyBuild/4.5.4/easybuild/reprod/easyblocks/pythonpackage.py'"), ('/tmp/eb-jy4z76eo/reprod_20220425030354_267603/easyblocks/easybuildmeta.py', '/panfs/users/Xcvleeu/icelake/software/EasyBuild/4.5.4/easybuild/reprod/easyblocks/easybuildmeta.py', "[Errno 22] Invalid argument: '/panfs/users/Xcvleeu/icelake/software/EasyBuild/4.5.4/easybuild/reprod/easyblocks/easybuildmeta.py'")]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/panfs/users/achaibi/surf_shared/icelake/software/EasyBuild/4.5.4/lib/python3.6/site-packages/easybuild/main.py", line 128, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/panfs/users/achaibi/surf_shared/icelake/software/EasyBuild/4.5.4/lib/python3.6/site-packages/easybuild/framework/easyblock.py", line 4140, in build_and_install_one
    raise error
  File "/panfs/users/achaibi/surf_shared/icelake/software/EasyBuild/4.5.4/lib/python3.6/site-packages/easybuild/framework/easyblock.py", line 4134, in build_and_install_one
    move_file(reprod_dir, archive_reprod_dir)
  File "/panfs/users/achaibi/surf_shared/icelake/software/EasyBuild/4.5.4/lib/python3.6/site-packages/easybuild/tools/filetools.py", line 2710, in move_file
    raise EasyBuildError("Failed to move %s to %s: %s", path, target_path, err)
easybuild.tools.build_log.EasyBuildError: 'Failed to move /tmp/eb-jy4z76eo/reprod_20220425030354_267603 to /panfs/users/Xcvleeu/icelake/software/EasyBuild/4.5.4/easybuild/reprod: [(\'/tmp/eb-jy4z76eo/reprod_20220425030354_267603/easyblocks/pythonpackage.py\', \'/panfs/users/Xcvleeu/icelake/software/EasyBuild/4.5.4/easybuild/reprod/easyblocks/pythonpackage.py\', "[Errno 22] Invalid argument: \'/panfs/users/Xcvleeu/icelake/software/EasyBuild/4.5.4/easybuild/reprod/easyblocks/pythonpackage.py\'"), (\'/tmp/eb-jy4z76eo/reprod_20220425030354_267603/easyblocks/easybuildmeta.py\', \'/panfs/users/Xcvleeu/icelake/software/EasyBuild/4.5.4/easybuild/reprod/easyblocks/easybuildmeta.py\', "[Errno 22] Invalid argument: \'/panfs/users/Xcvleeu/icelake/software/EasyBuild/4.5.4/easybuild/reprod/easyblocks/easybuildmeta.py\'")]'

[Xcvleeu@eia006 endeavour_sources]$ ls -al /panfs/users/Xcvleeu/icelake/software/EasyBuild/4.5.4/easybuild/reprod
total 208
drwxrwxr-x 3 Xcvleeu Xcvleeu 4096 Apr 25 03:09 .
drwxr-xr-x 5 Xcvleeu Xcvleeu 4096 Apr 25 03:09 ..
drwxrwxr-x 2 Xcvleeu Xcvleeu 4096 Apr 25 03:03 easyblocks
-rw-rw-r-- 1 Xcvleeu Xcvleeu 1778 Apr 25 03:03 EasyBuild-4.5.4.eb
-rw-rw-r-- 1 Xcvleeu Xcvleeu  248 Apr 25 03:09 EasyBuild-4.5.4.env

This stackoverflow suggested the issue was that an os.rename can fail if the source and target are on a different filesystem. To be honest, I'm not sure if that was the actual root cause: I think the Invalid argument in the second or third tracebacks (see folded summary above) might be the real causes.

Second issue: invalid argument
Because of the suggestion that the first issue could potentially be caused by the source and targets being on different filesystems, I tried to put the tempdir on the same /panfs filesystem and got my second issue.

== 2022-04-25 03:41:57,068 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/base/exceptions.py:124 in __init__): Failed to copy file /panfs/users/achaibi/surf_shared/icelake/software/EasyBuild/4.5.4/lib/python3.6/site-packages/easybuild/easyblocks/generic/pythonpackage.py to /panfs/users/Xcvleeu/eb_tmpdir/eb-bwve6dfs/reprod_20220425034157_267913/easyblocks/pythonpackage.py: [Errno 22] Invalid argument: '/panfs/users/Xcvleeu/eb_tmpdir/eb-bwve6dfs/reprod_20220425034157_267913/easyblocks/pythonpackage.py' (at easybuild/tools/filetools.py:2407 in copy_file)

Now, that looked strange: the file was actually copied to the tempdir:

ls -al /panfs/users/Xcvleeu/eb_tmpdir/eb-bwve6dfs/reprod_20220425034157_267913/easyblocks/
total 224
drwxrwxr-x 2 Xcvleeu Xcvleeu  4096 Apr 25 03:41 .
drwxrwxr-x 3 Xcvleeu Xcvleeu  4096 Apr 25 03:41 ..
-rw-r--r-- 1 Xcvleeu surf    45301 Mar 31 03:14 pythonpackage.py

yet still I got this error.

Root cause of the 2nd issue
With some help of @branfosj , I managed to trace down the real call that caused the Invalid argument issue. In filetools.py there is a try-except block here that calls an shutil.copy2 in the copy_file function. That's the one that really failed, but because the error is caught in the except, the actual stacktrace is not so clear. Running a new python shell, I tried:

>>> import shutil
>>> shutil.copy2("/panfs/users/Xcvleeu/icelake/software/EasyBuild/4.5.4/lib/python3.6/site-packa
ges/easybuild/easyblocks/generic/configuremake.py", "/panfs/users/Xcvleeu/eb-dwh1imvk/reprod_202
20425041120_277884/easyblocks/configuremake.py")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.6/shutil.py", line 264, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 229, in copystat
    _copyxattr(src, dst, follow_symlinks=follow)
  File "/usr/lib64/python3.6/shutil.py", line 165, in _copyxattr
    os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
OSError: [Errno 22] Invalid argument: '/panfs/users/Xcvleeu/eb-dwh1imvk/reprod_20220425041120_277884/easyblocks/configuremake.py'

Now, that's a more clear stacktrace. The issue is with os.setxattr and is probably the issue described here. When copying from NFS to NFS, this issue can pop up. It was probably resolved with python/cpython#13369, which was (if I understand correctly) backported to python 3.7, while I was using python 3.6.

Solution
Since I don't have administrative rights on this system, I could not update the system Python. However, there was a Python 2.7.18. Trying EB_PYTHON=python2, the both the first and the 2nd issue described above were resolved.

The issues are both the result of a bug in Python it seems, so we can probably not do much on the EasyBuild side. I have logged this issue and the potential solutions (update your Python 3, or try swapping to a Python 2 as long as that is still supported by EasyBuild) for future reference in case anyone else runs into this.