tox4: fails when requirements file contains a line a la `-e .[test] -c constraints.xt`

[jugmac00]

repo
pip install plone.recipe.zope2instance

tox.ini
https://github.com/plone/plone.recipe.zope2instance/blob/master/tox.ini

tox4 --version
4.0.0a6 from /home/jugmac00/.local/lib/python3.6/site-packages/tox/__init__.py
(oopsie, does that mean I did a `pip install --user` :-/ )

error possibly caused by the contents of the requirements file

https://github.com/plone/plone.recipe.zope2instance/blob/ec707d4fcb4ffed887ed3de20bfab626f6d56ef6/requirements-testing-zope-4.txt#L1-L5

❯ tox4 -rvv -e py38
ROOT: 148 D setup logging to DEBUG on pid 6564 [tox/report.py:211]
ROOT: 153 W remove tox env folder /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38 [tox/tox_env/api.py:202]
ROOT: 182 I find interpreter for spec PythonSpec(implementation=CPython, major=3, minor=8) [virtualenv/discovery/builtin.py:62]
ROOT: 182 I proposed PythonInfo(spec=CPython3.6.9.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.6.9 (default, Jan 26 2021, 15:33:00) \n[GCC 8.4.0]', encoding_fs_io=utf-8-UTF-8) [virtualenv/discovery/builtin.py:69]
ROOT: 183 D discover PATH[0]=/home/jugmac00/.cargo/bin [virtualenv/discovery/builtin.py:114]
ROOT: 183 D filesystem is case-sensitive [virtualenv/info.py:28]
ROOT: 184 D discover PATH[1]=/home/jugmac00/.local/bin [virtualenv/discovery/builtin.py:114]
ROOT: 184 D discover PATH[2]=/home/jugmac00/bin [virtualenv/discovery/builtin.py:114]
ROOT: 184 D discover PATH[3]=/usr/local/sbin [virtualenv/discovery/builtin.py:114]
ROOT: 184 D discover PATH[4]=/usr/local/bin [virtualenv/discovery/builtin.py:114]
ROOT: 184 D discover PATH[5]=/usr/sbin [virtualenv/discovery/builtin.py:114]
ROOT: 184 D discover PATH[6]=/usr/bin [virtualenv/discovery/builtin.py:114]
ROOT: 185 D Attempting to acquire lock 140009467336016 on /home/jugmac00/.local/share/virtualenv/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [filelock.py:270]
ROOT: 185 D Lock 140009467336016 acquired on /home/jugmac00/.local/share/virtualenv/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [filelock.py:274]
ROOT: 185 D got python info of /usr/bin/python3.8 from /home/jugmac00/.local/share/virtualenv/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.json [virtualenv/app_data/via_disk_folder.py:135]
ROOT: 186 D Attempting to release lock 140009467336016 on /home/jugmac00/.local/share/virtualenv/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [filelock.py:315]
ROOT: 186 D Lock 140009467336016 released on /home/jugmac00/.local/share/virtualenv/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [filelock.py:318]
ROOT: 186 I proposed PathPythonInfo(spec=CPython3.8.8.final.0-64, exe=/usr/bin/python3.8, platform=linux, version='3.8.8 (default, Feb 20 2021, 21:09:14) \n[GCC 7.5.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:69]
ROOT: 186 D accepted PathPythonInfo(spec=CPython3.8.8.final.0-64, exe=/usr/bin/python3.8, platform=linux, version='3.8.8 (default, Feb 20 2021, 21:09:14) \n[GCC 7.5.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:71]
ROOT: 211 I create virtual environment via CPython3Posix(dest=/home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38, clear=True, no_vcs_ignore=False, global=False) [virtualenv/run/session.py:52]
ROOT: 211 D delete /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38 [virtualenv/create/creator.py:169]
ROOT: 211 D create folder /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin [virtualenv/util/path/_sync.py:25]
ROOT: 211 D create folder /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages [virtualenv/util/path/_sync.py:25]
ROOT: 212 D write /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:34]
ROOT: 212 D 	home = /usr [virtualenv/create/pyenv_cfg.py:38]
ROOT: 212 D 	implementation = CPython [virtualenv/create/pyenv_cfg.py:38]
ROOT: 212 D 	version_info = 3.8.8.final.0 [virtualenv/create/pyenv_cfg.py:38]
ROOT: 212 D 	virtualenv = 20.4.2 [virtualenv/create/pyenv_cfg.py:38]
ROOT: 212 D 	include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:38]
ROOT: 212 D 	base-prefix = /usr [virtualenv/create/pyenv_cfg.py:38]
ROOT: 212 D 	base-exec-prefix = /usr [virtualenv/create/pyenv_cfg.py:38]
ROOT: 212 D 	base-executable = /usr/bin/python3.8 [virtualenv/create/pyenv_cfg.py:38]
ROOT: 212 D symlink /usr/bin/python3.8 to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin/python [virtualenv/util/path/_sync.py:44]
ROOT: 213 D create virtualenv import hook file /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/_virtualenv.pth [virtualenv/create/via_global_ref/api.py:95]
ROOT: 213 D create /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/_virtualenv.py [virtualenv/create/via_global_ref/api.py:98]
ROOT: 213 D ============================== target debug ============================== [virtualenv/run/session.py:54]
ROOT: 214 D debug via /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin/python /home/jugmac00/.local/lib/python3.6/site-packages/virtualenv/create/debug.py [virtualenv/create/creator.py:224]
ROOT: 213 D {
  "sys": {
    "executable": "/home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin/python",
    "_base_executable": "/home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin/python",
    "prefix": "/home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38",
    "base_prefix": "/usr",
    "real_prefix": null,
    "exec_prefix": "/home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib/python38.zip",
      "/usr/lib/python3.8",
      "/usr/lib/python3.8/lib-dynload",
      "/home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.8.8 (default, Feb 20 2021, 21:09:14) \n[GCC 7.5.0]",
  "makefile_filename": "/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' from '/usr/lib/python3.8/os.py'>",
  "site": "<module 'site' from '/usr/lib/python3.8/site.py'>",
  "datetime": "<module 'datetime' from '/usr/lib/python3.8/datetime.py'>",
  "math": "<module 'math' (built-in)>",
  "json": "<module 'json' from '/usr/lib/python3.8/json/__init__.py'>"
} [virtualenv/run/session.py:55]
ROOT: 255 I add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/jugmac00/.local/share/virtualenv) [virtualenv/run/session.py:59]
py38: 258 D got embed update of distribution setuptools from /home/jugmac00/.local/share/virtualenv/wheel/3.8/embed/1/setuptools.json [virtualenv/app_data/via_disk_folder.py:135]
py38: 258 D got embed update of distribution pip from /home/jugmac00/.local/share/virtualenv/wheel/3.8/embed/1/pip.json [virtualenv/app_data/via_disk_folder.py:135]
py38: 261 D got embed update of distribution wheel from /home/jugmac00/.local/share/virtualenv/wheel/3.8/embed/1/wheel.json [virtualenv/app_data/via_disk_folder.py:135]
py38: 262 D install setuptools from wheel /home/jugmac00/.local/lib/python3.6/site-packages/virtualenv/seed/wheels/embed/setuptools-52.0.0-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]
py38: 262 D install pip from wheel /home/jugmac00/.local/lib/python3.6/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]
py38: 262 D Attempting to acquire lock 140009440043920 on /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-52.0.0-py3-none-any.lock [filelock.py:270]
py38: 262 D Lock 140009440043920 acquired on /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-52.0.0-py3-none-any.lock [filelock.py:274]
py38: 263 D install wheel from wheel /home/jugmac00/.local/lib/python3.6/site-packages/virtualenv/seed/wheels/embed/wheel-0.36.2-py2.py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]
py38: 263 D Attempting to acquire lock 140009440192664 on /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.36.2-py2.py3-none-any.lock [filelock.py:270]
py38: 263 D Attempting to acquire lock 140009440045376 on /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-21.0.1-py3-none-any.lock [filelock.py:270]
py38: 263 D Attempting to release lock 140009440043920 on /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-52.0.0-py3-none-any.lock [filelock.py:315]
py38: 263 D Lock 140009440192664 acquired on /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.36.2-py2.py3-none-any.lock [filelock.py:274]
py38: 263 D Lock 140009440045376 acquired on /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-21.0.1-py3-none-any.lock [filelock.py:274]
py38: 263 D Lock 140009440043920 released on /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-52.0.0-py3-none-any.lock [filelock.py:318]
py38: 264 D Attempting to release lock 140009440192664 on /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.36.2-py2.py3-none-any.lock [filelock.py:315]
py38: 264 D Lock 140009440192664 released on /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.36.2-py2.py3-none-any.lock [filelock.py:318]
py38: 264 D Attempting to release lock 140009440045376 on /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-21.0.1-py3-none-any.lock [filelock.py:315]
py38: 264 D Lock 140009440045376 released on /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-21.0.1-py3-none-any.lock [filelock.py:318]
py38: 264 D copy /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.36.2-py2.py3-none-any/wheel-0.36.2.virtualenv to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/wheel-0.36.2.virtualenv [virtualenv/util/path/_sync.py:52]
py38: 264 D copy /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-52.0.0-py3-none-any/setuptools-52.0.0.virtualenv to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/setuptools-52.0.0.virtualenv [virtualenv/util/path/_sync.py:52]
py38: 264 D copy /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-21.0.1-py3-none-any/pip-21.0.1.virtualenv to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/pip-21.0.1.virtualenv [virtualenv/util/path/_sync.py:52]
py38: 265 D copy directory /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.36.2-py2.py3-none-any/wheel-0.36.2.dist-info to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/wheel-0.36.2.dist-info [virtualenv/util/path/_sync.py:52]
py38: 265 D copy directory /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-52.0.0-py3-none-any/pkg_resources to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/pkg_resources [virtualenv/util/path/_sync.py:52]
py38: 265 D copy directory /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-21.0.1-py3-none-any/pip-21.0.1.dist-info to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/pip-21.0.1.dist-info [virtualenv/util/path/_sync.py:52]
py38: 268 D copy directory /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.36.2-py2.py3-none-any/wheel to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/wheel [virtualenv/util/path/_sync.py:52]
py38: 268 D copy directory /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-21.0.1-py3-none-any/pip to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/pip [virtualenv/util/path/_sync.py:52]
py38: 276 D changing mode of /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin/wheel3 to 755 [distlib/util.py:566]
py38: 276 D changing mode of /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin/wheel3.8 to 755 [distlib/util.py:566]
py38: 276 D changing mode of /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin/wheel-3.8 to 755 [distlib/util.py:566]
py38: 277 D changing mode of /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin/wheel to 755 [distlib/util.py:566]
py38: 277 D generated console scripts wheel3.8 wheel3 wheel-3.8 wheel [virtualenv/seed/embed/via_app_data/pip_install/base.py:48]
py38: 277 D copy /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-52.0.0-py3-none-any/distutils-precedence.pth to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/distutils-precedence.pth [virtualenv/util/path/_sync.py:52]
py38: 277 D copy directory /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-52.0.0-py3-none-any/_distutils_hack to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/_distutils_hack [virtualenv/util/path/_sync.py:52]
py38: 278 D copy directory /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-52.0.0-py3-none-any/setuptools-52.0.0.dist-info to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/setuptools-52.0.0.dist-info [virtualenv/util/path/_sync.py:52]
py38: 280 D copy directory /home/jugmac00/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-52.0.0-py3-none-any/setuptools to /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/lib/python3.8/site-packages/setuptools [virtualenv/util/path/_sync.py:52]
py38: 315 D generated console scripts  [virtualenv/seed/embed/via_app_data/pip_install/base.py:48]
py38: 346 D changing mode of /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin/pip3.8 to 755 [distlib/util.py:566]
py38: 346 D changing mode of /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin/pip-3.8 to 755 [distlib/util.py:566]
py38: 346 D changing mode of /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin/pip to 755 [distlib/util.py:566]
py38: 347 D changing mode of /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/bin/pip3 to 755 [distlib/util.py:566]
py38: 347 D generated console scripts pip3.8 pip-3.8 pip3 pip [virtualenv/seed/embed/via_app_data/pip_install/base.py:48]
ROOT: 347 I add activators for Bash, CShell, Fish, PowerShell, Python, Xonsh [virtualenv/run/session.py:66]
ROOT: 348 D write /home/jugmac00/Projects/plone.recipe.zope2instance/.tox/4/py38/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:34]
ROOT: 348 D 	home = /usr [virtualenv/create/pyenv_cfg.py:38]
ROOT: 348 D 	implementation = CPython [virtualenv/create/pyenv_cfg.py:38]
ROOT: 348 D 	version_info = 3.8.8.final.0 [virtualenv/create/pyenv_cfg.py:38]
ROOT: 348 D 	virtualenv = 20.4.2 [virtualenv/create/pyenv_cfg.py:38]
ROOT: 348 D 	include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:38]
ROOT: 348 D 	base-prefix = /usr [virtualenv/create/pyenv_cfg.py:38]
ROOT: 348 D 	base-exec-prefix = /usr [virtualenv/create/pyenv_cfg.py:38]
ROOT: 348 D 	base-executable = /usr/bin/python3.8 [virtualenv/create/pyenv_cfg.py:38]
ROOT: 350 E internal error [tox/session/cmd/run/single.py:53]
Traceback (most recent call last):
  File "/home/jugmac00/.local/lib/python3.6/site-packages/tox/session/cmd/run/single.py", line 41, in _evaluate
    tox_env.ensure_setup(recreate=recreate)
  File "/home/jugmac00/.local/lib/python3.6/site-packages/tox/tox_env/api.py", line 179, in ensure_setup
    self.setup()
  File "/home/jugmac00/.local/lib/python3.6/site-packages/tox/tox_env/python/virtual_env/api.py", line 109, in setup
    super().setup()
  File "/home/jugmac00/.local/lib/python3.6/site-packages/tox/tox_env/python/api.py", line 162, in setup
    super().setup()
  File "/home/jugmac00/.local/lib/python3.6/site-packages/tox/tox_env/runner.py", line 81, in setup
    self.before_package_install()
  File "/home/jugmac00/.local/lib/python3.6/site-packages/tox/tox_env/python/runner.py", line 49, in before_package_install
    requirement_file_content = requirements_file.validate_and_expand()
  File "/home/jugmac00/.local/lib/python3.6/site-packages/tox/tox_env/python/req_file.py", line 113, in validate_and_expand
    self._expand_flag(ini_dir, line, result)
  File "/home/jugmac00/.local/lib/python3.6/site-packages/tox/tox_env/python/req_file.py", line 165, in _expand_flag
    result.extend(req_file.validate_and_expand())
  File "/home/jugmac00/.local/lib/python3.6/site-packages/tox/tox_env/python/req_file.py", line 113, in validate_and_expand
    self._expand_flag(ini_dir, line, result)
  File "/home/jugmac00/.local/lib/python3.6/site-packages/tox/tox_env/python/req_file.py", line 155, in _expand_flag
    raise ValueError(line)
ValueError: -e .[test] -c https://zopefoundation.github.io/Zope/releases/4.5/requirements-full.txt
  py38: FAIL code 2 (0.20 seconds)
  evaluation failed :( (0.23 seconds)

[gaborbernat]

-e .[test] -c https://zopefoundation.github.io/Zope/releases/4.5.4/requirements-full.txt

# Windows specific down here (has to be installed here, fails in buildout)
# Dependency of zope.sendmail:
pywin32 ; platform_system == 'Windows'

This is unsupported (at the moment) for two reasons:

• multiple arguments per line in requirements.txt
• external (HTTPS) constraint file

[jugmac00]

This one is unaffected by #1991 and remains open.

[gaborbernat]

So I talked with Pradyung about this, and it seems specifying a constraint file on the same line as an editable expression is something that works by chance rather than by design, and he'd deprecate it than support it. That being said, I've realized that we don't currently support per requirement flags, so the requirements.txt file parser does need more love. On the plus side, if we implement a parser pip would be open to using it.

[gaborbernat]

So after looking into this, technically tox here fails, but does the right thing; because of this line:

-e .[test] -c https://zopefoundation.github.io/Zope/releases/4.5.4/requirements-full.txt

is invalid. You cannot have two options on the same line. pip does not fail but is ignoring the constraint file... https://pip--9474.org.readthedocs.build/en/9474/reference/requirements-files/#supported-options documents that you must have one option/requirement per line. Though pip never validates this, just silently ignores the rest found.

[jugmac00]

Thanks a lot for investigating this problem!

I created a pull request for above shown requirements.txt ( plone/plone.recipe.zope2instance#172 ).

What is left to do here? Do you prefer to leave it here as it is, or do you prefer to catch the error and present a more friendly message to the end user?

[gaborbernat]

We should add a FAQ page to tox 4 and document my solution, when using constraint files.

[jugmac00]

Meanwhile, there is a FAQ entry which documents how to use constraints files with tox -> closing this issue.

https://tox.readthedocs.io/en/rewrite/faq.html#using-constraint-files