Narrow type variable bounds in binder

[ilevkivskyi]

Fixes #5720
Fixes #8556
Fixes #9778
Fixes #10003
Fixes #10817
Fixes #11163
Fixes #11664
Fixes #12882
Fixes #13426
Fixes #13462
Fixes #14941
Fixes #15151
Fixes #19166

This handles a (surprisingly) common edge case. The charges in bind_self() and bind_self_fast() are tricky. I got a few "Redundant cast" errors there, which seemed good, but then I realized that attribute access etc. on a type variable go through slow PyObject paths, so I am actually forcing CallableType instead of F(bound=CallableType) there, since these are performance-critical functions.

[ilevkivskyi]

OK, the mypy_primer results are good, and although it looks like a niche use case, this PR fixes more than dozen issues (some with many upvotes).

[hauntsaninja]

Hooray!

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

PyGithub (https://github.com/PyGithub/PyGithub)
+ github/MainClass.py:988: error: Unused "type: ignore" comment  [unused-ignore]

antidote (https://github.com/Finistere/antidote): 1.47x slower (65.5s -> 96.3s in single noisy sample)

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/states.py:369: error: Argument 1 to "StateGroup" has incompatible type "Collection[Union[State[Any], Any]]"; expected "list[State[Any]]"  [arg-type]
+ src/prefect/states.py:369: error: Argument 1 to "StateGroup" has incompatible type "Collection[Never]"; expected "list[State[Any]]"  [arg-type]
+ src/prefect/futures.py:222: note:     def result(self, raise_on_failure: Literal[True] = ..., retry_result_failure: bool = ...) -> Any
+ src/prefect/futures.py:222: note:     def result(self, raise_on_failure: Literal[False] = ..., retry_result_failure: bool = ...) -> Union[Any, Exception]
+ src/prefect/futures.py:222: note:     def result(self, raise_on_failure: bool = ..., retry_result_failure: bool = ...) -> Union[Any, Exception]
- src/prefect/task_engine.py:467: error: Item "type" of "Union[ResultRecord[Any], type[NotSet]]" has no attribute "result"  [union-attr]
+ src/prefect/task_engine.py:467: error: Item "type" of "Union[R, type[NotSet]]" has no attribute "result"  [union-attr]
- src/prefect/task_engine.py:1022: error: Item "type" of "Union[ResultRecord[Any], type[NotSet]]" has no attribute "result"  [union-attr]
+ src/prefect/task_engine.py:1022: error: Item "type" of "Union[R, type[NotSet]]" has no attribute "result"  [union-attr]
- src/prefect/flow_runs.py:292: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "load"  [union-attr]
+ src/prefect/flow_runs.py:292: error: "type[T]" has no attribute "load"  [attr-defined]
- src/prefect/flow_runs.py:307: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "save"  [union-attr]
+ src/prefect/flow_runs.py:307: error: "type[T]" has no attribute "save"  [attr-defined]
- src/prefect/flow_runs.py:319: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "load"  [union-attr]
+ src/prefect/flow_runs.py:319: error: "type[T]" has no attribute "load"  [attr-defined]
- src/prefect/flow_runs.py:446: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "load"  [union-attr]
+ src/prefect/flow_runs.py:446: error: "type[T]" has no attribute "load"  [attr-defined]
- src/prefect/flow_runs.py:456: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "save"  [union-attr]
+ src/prefect/flow_runs.py:456: error: "type[T]" has no attribute "save"  [attr-defined]

kopf (https://github.com/nolar/kopf)
+ kopf/_cogs/structs/dicts.py:243: error: Unused "type: ignore" comment  [unused-ignore]
+ kopf/_core/intents/handlers.py:36: error: Redundant cast to "execution.CauseT"  [redundant-cast]

python-chess (https://github.com/niklasf/python-chess)
- chess/pgn.py:1315: error: Returning Any from function declared to return "HeadersT"  [no-any-return]

kornia (https://github.com/kornia/kornia)
+ kornia/utils/helpers.py:387: error: Unused "type: ignore" comment  [unused-ignore]

mypy (https://github.com/python/mypy)
+ mypy/expandtype.py:125: error: Redundant cast to "F"  [redundant-cast]
+ mypy/expandtype.py:125: note: See https://mypy.rtfd.io/en/stable/_refs.html#code-redundant-cast for more info
+ mypy/typeops.py:421: error: Redundant cast to "F"  [redundant-cast]
+ mypy/typeops.py:430: error: Redundant cast to "F"  [redundant-cast]
+ mypy/typeops.py:469: error: Incompatible types in assignment (expression has type "CallableType", variable has type "F")  [assignment]
+ mypy/typeops.py:484: error: Redundant cast to "F"  [redundant-cast]
+ mypy/checkmember.py:1489: error: Redundant cast to "F"  [redundant-cast]
+ mypy/checkmember.py:1492: error: Redundant cast to "F"  [redundant-cast]
+ mypy/checkmember.py:1502: error: Redundant cast to "F"  [redundant-cast]

hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- src/hydra_zen/third_party/beartype.py:128: error: Incompatible types in assignment (expression has type "_T", variable has type "type[Any]")  [assignment]
+ src/hydra_zen/third_party/beartype.py:128: error: Incompatible types in assignment (expression has type "_T", variable has type "_T")  [assignment]
+ src/hydra_zen/third_party/beartype.py:130: error: Redundant cast to "_T"  [redundant-cast]
+ src/hydra_zen/third_party/pydantic.py:111: error: Redundant cast to "_T"  [redundant-cast]
+ src/hydra_zen/third_party/pydantic.py:115: error: Redundant cast to "_T"  [redundant-cast]
- src/hydra_zen/third_party/pydantic.py:123: error: Argument 1 has incompatible type "_T | type[object]"; expected "_T"  [arg-type]

pytest (https://github.com/pytest-dev/pytest)
- src/_pytest/fixtures.py:1132: error: Returning Any from function declared to return "FixtureValue"  [no-any-return]

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
+ tests/__init__.py:67: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/__init__.py:85: error: Unused "type: ignore" comment  [unused-ignore]

beartype (https://github.com/beartype/beartype)
+ beartype/_decor/_type/decortype.py:133: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_type/decortype.py:295: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:294: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:301: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:380: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:395: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:567: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/decorcore.py:133: error: Unused "type: ignore" comment  [unused-ignore]

xarray (https://github.com/pydata/xarray)
+ xarray/structure/alignment.py: note: In function "_broadcast_helper":
+ xarray/structure/alignment.py:1140: error: Redundant cast to "T_Alignable"  [redundant-cast]
+ xarray/structure/alignment.py:1142: error: Redundant cast to "T_Alignable"  [redundant-cast]

steam.py (https://github.com/Gobot1234/steam.py)
- steam/ext/commands/commands.py:284: error: Argument 1 to "append" of "list" has incompatible type "Command[Any, Any, Any]"; expected "Self"  [arg-type]
- steam/ext/commands/cog.py:182: error: Argument 1 to "decorator" has incompatible type "Callable[[Any | None, VarArg(Any), KwArg(Any)], Coroutine[Any, Any, None]]"; expected "F"  [arg-type]
- steam/ext/commands/bot.py:415: error: Argument 1 to "decorator" has incompatible type "Callable[..., Coroutine[Any, Any, Any]]"; expected "F"  [arg-type]

[netomi]

ftr: this commit fixes a bug we encountered with mypy 1.16.0 (and not in 1.15.0) in different places with a stacktrace like:

asfyaml/feature/github/pull_requests.py: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.16.0
Traceback (most recent call last):
  File "/home/tn/workspace/apache/infrastructure-asfyaml/.venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "mypy/build.py", line 2062, in wrap_context
RuntimeError: ('Not implemented', <class 'mypy.types.TypeGuardedType'>)
asfyaml/feature/github/pull_requests.py: : note: use --pdb to drop into pdb

[ilevkivskyi]

@netomi Hm, this is interesting, could you please post a full traceback? (use --show-traceback)

[netomi]

that is already with this option:

tn@proteus:~/workspace/apache/infrastructure-asfyaml$ poetry run mypy --show-traceback asfyaml/feature/github/pull_requests.py 
asfyaml/feature/github/pull_requests.py: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.16.0
Traceback (most recent call last):
  File "/home/tn/workspace/apache/infrastructure-asfyaml/.venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "mypy/build.py", line 2062, in wrap_context
RuntimeError: ('Not implemented', <class 'mypy.types.TypeGuardedType'>)
asfyaml/feature/github/pull_requests.py: : note: use --pdb to drop into pdb

the project can be found here: https://github.com/apache/infrastructure-asfyaml

it is now using 1.15.0, but updating to 1.16.0 shows the bug.

[ilevkivskyi]

A bit more of the traceback

  File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/build.py", line 2062, in wrap_context
    yield
  File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/build.py", line 2391, in finish_passes
    self._patch_indirect_dependencies(self.type_checker().module_refs, all_types)
  File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/build.py", line 2424, in _patch_indirect_dependencies
    encountered = self.manager.indirection_detector.find_modules(types) | module_refs
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/indirection.py", line 35, in find_modules
    self._visit(typs)
  File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/indirection.py", line 46, in _visit
    typ.accept(self)
  File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/types.py", line 276, in accept
    raise RuntimeError("Not implemented", type(self))
RuntimeError: ('Not implemented', <class 'mypy.types.TypeGuardedType'>)

This looks like something where I would say this PR may be indeed a proper fix. So looks like no action needed (apart from releasing 1.17 LOL)

[netomi]

just wanted to mention it as there might be others that report issues about that, so there should be a changelog entry to track that this is solved in 1.17.0 imho.