pipenv does not pull in extras if extra name does not match module name

[thehesiod]

Issue description

If a requirement has extras, with a name that does not match the module name the module will not get added to the lockfile

$ cat Pipfile
[packages]
hypothesis = {extras = ["datetime"]}

[requires]
python_version = "3.11"

Note that the datetime extra should pull in pytz: https://github.com/bennylope/hypothesis-python/blob/master/setup.py#L41

Expected result

Extras specified should get added

Actual result

$ cat Pipfile.lock 
{
    "_meta": {
        "hash": {
            "sha256": "f1eb1944c7bc948a5bc4e1d5dd0a8e75b99e6b43cad9d3a3c222e3f840a9a3bc"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.11"
        },
        "sources": [
            {
                "name": "pip_conf_index_global",
                "url": "https://pypi.fbn.org/simple",
                "verify_ssl": false
            }
        ]
    },
    "default": {
        "attrs": {
            "hashes": [
                "sha256:1f28b4522cdc2fb4256ac1a020c78acf9cba2c6b461ccd2c126f3aa8e8335d04",
                "sha256:6279836d581513a26f1bf235f9acd333bc9115683f14f7e8fae46c98fc50e015"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==23.1.0"
        },
        "hypothesis": {
            "extras": [
                "datetime"
            ],
            "hashes": [
                "sha256:e1d36522824d62bb3e9fcb7b57dd4a6ca330bb36921324bb19c476bdafabeda7",
                "sha256:e5d75d70f5a4fc372cddf03ec6141237a0a270ed106aeb2156a4984f06d37b0f"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==6.86.2"
        },
        "sortedcontainers": {
            "hashes": [
                "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88",
                "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0"
            ],
            "version": "==2.4.0"
        }
    },
    "develop": {}
}

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

---

$ pipenv --support

Pipenv version: '2023.9.8'

Pipenv location: '/opt/homebrew/lib/python3.11/site-packages/pipenv'

Python location: '/opt/homebrew/opt/python@3.11/bin/python3.11'

OS Name: 'posix'

User pip version: '23.2.1'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.11.4',
 'os_name': 'posix',
 'platform_machine': 'arm64',
 'platform_python_implementation': 'CPython',
 'platform_release': '22.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 22.6.0: Fri Sep 15 13:41:28 PDT '
                     '2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T6000',
 'python_full_version': '3.11.4',
 'python_version': '3.11',
 'sys_platform': 'darwin'}

System environment variables:

• SHELL
• NVM_RC_VERSION
• XPC_FLAGS
• HISTCONTROL
• TERM_PROGRAM_VERSION
• PKG_CONFIG_PATH
• HISTSIZE
• __CFBundleIdentifier
• SSH_AUTH_SOCK
• TERM_SESSION_ID
• GPG_TTY
• PWD
• LOGNAME
• LaunchInstanceID
• HOME
• LANG
• SECURITYSESSIONID
• TMPDIR
• FBNSECRETS_FORCE_USER
• NVM_DIR
• GEM_HOME
• TERM
• USER
• SHLVL
• NVM_CD_FLAGS
• XPC_SERVICE_NAME
• PS1
• PATH
• HOMEBREW_CASK_OPTS
• OLDPWD
• TERM_PROGRAM
• _
• __CF_USER_TEXT_ENCODING
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /Users/alexmohr/Downloads/google-cloud-sdk/bin:/opt/homebrew/opt/openjdk/bin:/Users/alexmohr/.gem/bin:/opt/homebrew/bin:/opt/homebrew/opt/openssl@1.1/bin:/opt/homebrew/opt/python@3.8/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/homebrew/bin:/Library/Apple/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/homebrew/opt/openjdk/bin:~/.gem/ruby/2.6.0/bin:/Users/alexmohr/dev/fbn.com/ops/docker/ecr/bin:~/Applications/Sublime Text.app/Contents/SharedSupport/bin
• SHELL: /opt/homebrew/bin/bash
• LANG: en_US.UTF-8
• PWD: /tmp/test

---

Contents of Pipfile ('/private/tmp/test/Pipfile'):

[packages]
hypothesis = {extras = ["datetime"]}

[requires]
python_version = "3.11"

Contents of Pipfile.lock ('/private/tmp/test/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "f1eb1944c7bc948a5bc4e1d5dd0a8e75b99e6b43cad9d3a3c222e3f840a9a3bc"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.11"
        },
        "sources": [
            {
                "name": "pip_conf_index_global",
                "url": "https://pypi.fbn.org/simple",
                "verify_ssl": false
            }
        ]
    },
    "default": {
        "attrs": {
            "hashes": [
                "sha256:1f28b4522cdc2fb4256ac1a020c78acf9cba2c6b461ccd2c126f3aa8e8335d04",
                "sha256:6279836d581513a26f1bf235f9acd333bc9115683f14f7e8fae46c98fc50e015"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==23.1.0"
        },
        "hypothesis": {
            "extras": [
                "datetime"
            ],
            "hashes": [
                "sha256:e1d36522824d62bb3e9fcb7b57dd4a6ca330bb36921324bb19c476bdafabeda7",
                "sha256:e5d75d70f5a4fc372cddf03ec6141237a0a270ed106aeb2156a4984f06d37b0f"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==6.86.2"
        },
        "sortedcontainers": {
            "hashes": [
                "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88",
                "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0"
            ],
            "version": "==2.4.0"
        }
    },
    "develop": {}
}

[matteius]

We need the proper package name for pip resolve/install, that is why.

[thehesiod]

ya but this works in pipenv 2023.7.23 so this is a regression

[matteius]

It can be considered a regression, but its not one I intend to "fix" this since was dependent on requirementslib behavior which has since been removed. I don't think its unreasonable to derive the package name from the Pipfile entry -- trying to determine it otherwise from the package itself is an expensive and error prone operation especially considering the many forms of setup.py and the desire to not evaluate code from setup.py during package resolution. Open to patches, but I won't be pursuing a patch for this.

[thehesiod]

We need the proper package name for pip resolve/install, that is why.

I'm not sure what you mean by you need the proper package name. extras_require is a thing: https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies . The top level key is an alias to one of more real dependencies. From what I see pipenv will pull the req if the alias is the name of a package, but won't dig into the actual requirements for the extra. Definitely seems like a bug. The hypothesis boils down to:

extras_require={
    'datetime':  ["pytz"],
    'fakefactory': ["Faker>=0.7.0,<=0.7.1"],
    'django': ['pytz', 'django>=1.8,<2,!=1.9.*'],
    'numpy': ['numpy>=1.9.0'],
    'pytest': ['pytest>=2.8.0'],
}

if they instead put:

extras_require={
    'pytz':  ["pytz"],

and in the Pipfile we put:

hypothesis = {extras = ["pytz"]}

Remember we need to specify the alias because the alias can point to one or more modules.

pipenv would pull the requirement. This needs to get fixed otherwise we cannot rely on pipenv properly enforcing version requirements across top level modules, and extras.

[thehesiod]

the original Pipfile is equivalent to pip install hypothesis[datetime], which would follow the alias and also install pytz

[matteius]

Sorry if there was a misunderstanding, I thought you were saying the name of the requirement in the Pipfile did not match the requirement name. When I try your example the resolve prints out:

[ ] Locking...Warning: WARNING:pipenv.patched.pip._internal.resolution.resolvelib.candidates:hypothesis 6.86.2 does not provide the extra 'datetime'

When I checked the wheel for hypothesis, we can see there is no extras group named datetime in the published wheel. It is named pytz there (I am not sure why). That is what the pypi resolution will use the wheel metadata to determine the extras, so I don't think that is a pipenv bug.

Metadata-Version: 2.1
Name: hypothesis
Version: 6.86.2
Summary: A library for property-based testing
Home-page: https://hypothesis.works
Author: David R. MacIver and Zac Hatfield-Dodds
Author-email: david@drmaciver.com
License: MPL-2.0
Project-URL: Source, https://github.com/HypothesisWorks/hypothesis/tree/master/hypothesis-python
Project-URL: Changelog, https://hypothesis.readthedocs.io/en/latest/changes.html
Project-URL: Documentation, https://hypothesis.readthedocs.io
Project-URL: Issues, https://github.com/HypothesisWorks/hypothesis/issues
Keywords: python testing fuzzing property-based-testing
Classifier: Development Status :: 5 - Production/Stable
Classifier: Framework :: Hypothesis
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)
Classifier: Operating System :: Unix
Classifier: Operating System :: POSIX
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Education :: Testing
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.8
Description-Content-Type: text/x-rst
License-File: LICENSE.txt
Requires-Dist: attrs >=19.2.0
Requires-Dist: sortedcontainers <3.0.0,>=2.1.0
Requires-Dist: exceptiongroup >=1.0.0 ; python_version < "3.11"
Provides-Extra: all
Requires-Dist: black >=19.10b0 ; extra == 'all'
Requires-Dist: click >=7.0 ; extra == 'all'
Requires-Dist: django >=3.2 ; extra == 'all'
Requires-Dist: dpcontracts >=0.4 ; extra == 'all'
Requires-Dist: lark >=0.10.1 ; extra == 'all'
Requires-Dist: libcst >=0.3.16 ; extra == 'all'
Requires-Dist: numpy >=1.17.3 ; extra == 'all'
Requires-Dist: pandas >=1.1 ; extra == 'all'
Requires-Dist: pytest >=4.6 ; extra == 'all'
Requires-Dist: python-dateutil >=1.4 ; extra == 'all'
Requires-Dist: pytz >=2014.1 ; extra == 'all'
Requires-Dist: redis >=3.0.0 ; extra == 'all'
Requires-Dist: rich >=9.0.0 ; extra == 'all'
Requires-Dist: backports.zoneinfo >=0.2.1 ; (python_version < "3.9") and extra == 'all'
Requires-Dist: tzdata >=2023.3 ; (sys_platform == "win32") and extra == 'all'
Provides-Extra: cli
Requires-Dist: click >=7.0 ; extra == 'cli'
Requires-Dist: black >=19.10b0 ; extra == 'cli'
Requires-Dist: rich >=9.0.0 ; extra == 'cli'
Provides-Extra: codemods
Requires-Dist: libcst >=0.3.16 ; extra == 'codemods'
Provides-Extra: dateutil
Requires-Dist: python-dateutil >=1.4 ; extra == 'dateutil'
Provides-Extra: django
Requires-Dist: django >=3.2 ; extra == 'django'
Provides-Extra: dpcontracts
Requires-Dist: dpcontracts >=0.4 ; extra == 'dpcontracts'
Provides-Extra: ghostwriter
Requires-Dist: black >=19.10b0 ; extra == 'ghostwriter'
Provides-Extra: lark
Requires-Dist: lark >=0.10.1 ; extra == 'lark'
Provides-Extra: numpy
Requires-Dist: numpy >=1.17.3 ; extra == 'numpy'
Provides-Extra: pandas
Requires-Dist: pandas >=1.1 ; extra == 'pandas'
Provides-Extra: pytest
Requires-Dist: pytest >=4.6 ; extra == 'pytest'
Provides-Extra: pytz
Requires-Dist: pytz >=2014.1 ; extra == 'pytz'
Provides-Extra: redis
Requires-Dist: redis >=3.0.0 ; extra == 'redis'
Provides-Extra: zoneinfo
Requires-Dist: backports.zoneinfo >=0.2.1 ; (python_version < "3.9") and extra == 'zoneinfo'
Requires-Dist: tzdata >=2023.3 ; (sys_platform == "win32") and extra == 'zoneinfo'

When I try with the git repository, I have to manually manipulate the Pipfile entry to have the extras but that is working:

$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "a53d9843a7d846384f64ea100fdda25f9358f7d2151b5276169fe7adb7deea3f"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.9"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "hypothesis": {
            "extras": [
                "datetime"
            ],
            "git": "git+https://github.com/bennylope/hypothesis-python.git",
            "markers": "python_version >= '3.8'",
            "ref": "73ad1f3b0ac1da959134b5a9f87659a2b3408c34"
        },
        "pytz": {
            "hashes": [
                "sha256:7b4fddbeb94a1eba4b557da24f19fdf9db575192544270a9101d8509f9f43d7b",
                "sha256:ce42d816b81b68506614c11e8937d3aa9e41007ceb50bfdcb0749b921bf646c7"
            ],
            "version": "==2023.3.post1"
        }
    },
    "develop": {}
}

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/issue-5947
$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
hypothesis = {ref = "master", git = "git+https://github.com/bennylope/hypothesis-python.git", extras=["datetime"]}

[dev-packages]

[requires]
python_version = "3.9"

The biggest issue I can see is that there should be a way to pipenv install the Pipfile entry with the extras for the git repository example without having to edit the Pipfile. However for the case of the wheels extras groups not matching the setup.py extras groups from 8 years ago -- probably has to do with a recently published wheel of the hypothesis project?

[thehesiod]

ok perhaps I picked a bad example, we're seeing this same issue at work for our own module, let me make an example w/o pypi

[thehesiod]

sorry, will close this until I get a test case that buttons down all the hatches :)