Skip to content

Commit 5324bd0

Browse files
hauntsaninjahauntsaninja
authored
typeshed: run stubtest in CI (#3727)
* README.md: refactor "Running the tests" This organises the section a little better. Previously some tests were unmentioned; it read as if mypy_test and pytype_test were the only tests. The section is now organised by test, making it easy to keep track of the requirements and details of each. This also makes it easier to add documentation for stubtest. Also mention turning on Travis CI on your fork, since that is very useful. * README.md: document stubtest_test.py * stubtest_test: add it * travis: add stubtest_test to CI * stubtest_test: add whitelists
1 parent 9ec0bcf commit 5324bd0

File tree

8 files changed

+1765
-34
lines changed

8 files changed

+1765
-34
lines changed

‎.travis.yml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,3 +29,19 @@ jobs:
2929
- name: "flake8"
3030
install: pip install -r requirements-tests-py3.txt
3131
script: flake8
32+
- name: "stubtest py38"
33+
python: 3.8
34+
install: pip install -U git+git://github.com/python/mypy@a07dbd00
35+
script: ./tests/stubtest_test.py
36+
- name: "stubtest py37"
37+
python: 3.7
38+
install: pip install -U git+git://github.com/python/mypy@a07dbd00
39+
script: ./tests/stubtest_test.py
40+
- name: "stubtest py36"
41+
python: 3.6
42+
install: pip install -U git+git://github.com/python/mypy@a07dbd00
43+
script: ./tests/stubtest_test.py
44+
- name: "stubtest py35"
45+
python: 3.5
46+
install: pip install -U git+git://github.com/python/mypy@a07dbd00
47+
script: ./tests/stubtest_test.py

‎README.md‎

Lines changed: 75 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -86,24 +86,22 @@ requests. If you have questions related to contributing, drop by the [typing Git
8686
## Running the tests
8787

8888
The tests are automatically run by Travis CI on every PR and push to
89-
the repo. There are several sets of tests: `tests/mypy_test.py`
90-
runs tests against [mypy](https://github.com/python/mypy/), while
91-
`tests/pytype_test.py` runs tests against
92-
[pytype](https://github.com/google/pytype/).
93-
94-
Both sets of tests are shallow -- they verify that all stubs can be
95-
imported but they don't check whether stubs match their implementation
96-
(in the Python standard library or a third-party package). Also note
97-
that each set of tests has a blacklist of modules that are not tested
98-
at all. The blacklists also live in the tests directory.
89+
the repo. Note that it can be useful to enable Travis CI on your own fork of
90+
typeshed.
9991

100-
In addition, you can run `tests/mypy_selftest.py` to run mypy's own
101-
test suite using the typeshed code in your repo. This will sometimes
102-
catch issues with incorrectly typed stubs, but is much slower than the
103-
other tests.
92+
There are several tests:
93+
- `tests/mypy_test.py`
94+
runs tests against [mypy](https://github.com/python/mypy/)
95+
- `tests/pytype_test.py` runs tests against
96+
[pytype](https://github.com/google/pytype/).
97+
- `tests/mypy_selftest.py` runs mypy's test suite using this version of
98+
typeshed.
99+
- `tests/check_consistent.py` checks certain files in typeshed remain
100+
consistent with each other.
101+
- `tests/stubtest_test.py` checks stubs against the objects at runtime.
102+
- `flake8` enforces a style guide.
104103

105-
To manually run the mypy tests, you need to have Python 3.5 or higher;
106-
Python 3.6.1 or higher is recommended.
104+
### Setup
107105

108106
Run:
109107
```
@@ -112,31 +110,74 @@ $ source .venv3/bin/activate
112110
(.venv3)$ pip3 install -r requirements-tests-py3.txt
113111
```
114112
This will install mypy (you need the latest master branch from GitHub),
115-
typed-ast, flake8, and pytype. You can then run mypy, flake8, and pytype tests
116-
by invoking:
117-
```
118-
(.venv3)$ python3 tests/mypy_test.py
119-
...
120-
(.venv3)$ python3 tests/mypy_selftest.py
121-
...
122-
(.venv3)$ flake8
123-
...
124-
(.venv3)$ python3 tests/pytype_test.py
125-
...
126-
```
127-
Note that flake8 only works with Python 3.6 or higher, and that to run the
128-
pytype tests, you will need Python 2.7 and Python 3.6 interpreters. Pytype will
129-
find these automatically if they're in `PATH`, but otherwise you must point to
130-
them with the `--python27-exe` and `--python36-exe` arguments, respectively.
113+
typed-ast, flake8 (and plugins), pytype, black and isort.
114+
115+
### mypy_test.py
131116

132-
For mypy, if you are in the typeshed repo that is submodule of the
117+
This test requires Python 3.5 or higher; Python 3.6.1 or higher is recommended.
118+
Run using:`(.venv3)$ python3 tests/mypy_test.py`
119+
120+
This test is shallow — it verifies that all stubs can be
121+
imported but doesn't check whether stubs match their implementation
122+
(in the Python standard library or a third-party package). It has a blacklist of
123+
modules that are not tested at all, which also lives in the tests directory.
124+
125+
If you are in the typeshed repo that is submodule of the
133126
mypy repo (so `..` refers to the mypy repo), there's a shortcut to run
134127
the mypy tests that avoids installing mypy:
135128
```bash
136129
$ PYTHONPATH=../.. python3 tests/mypy_test.py
137130
```
138-
You can mypy tests to a single version by passing `-p2` or `-p3.5` e.g.
131+
You can restrict mypy tests to a single version by passing `-p2` or `-p3.5`:
139132
```bash
140133
$ PYTHONPATH=../.. python3 tests/mypy_test.py -p3.5
141134
running mypy --python-version 3.5 --strict-optional # with 342 files
142135
```
136+
137+
### pytype_test.py
138+
139+
This test requires Python 2.7 and Python 3.6. Pytype will
140+
find these automatically if they're in `PATH`, but otherwise you must point to
141+
them with the `--python27-exe` and `--python36-exe` arguments, respectively.
142+
Run using: `(.venv3)$ python3 tests/pytype_test.py`
143+
144+
This test works similarly to `mypy_test.py`, except it uses `pytype`.
145+
146+
### mypy_selftest.py
147+
148+
This test requires Python 3.5 or higher; Python 3.6.1 or higher is recommended.
149+
Run using: `(.venv3)$ python3 tests/mypy_selftest.py`
150+
151+
This test runs mypy's own test suite using the typeshed code in your repo. This
152+
will sometimes catch issues with incorrectly typed stubs, but is much slower
153+
than the other tests.
154+
155+
### check_consistent.py
156+
157+
Run using: `python3 tests/check_consistent.py`
158+
159+
### stubtest_test.py
160+
161+
This test requires Python 3.5 or higher.
162+
Run using `(.venv3)$ python3 tests/stubtest_test.py`
163+
164+
This test compares the stdlib stubs against the objects at runtime. Because of
165+
this, the output depends on which version of Python it is run with.
166+
If you need a specific version of Python to repro a CI failure,
167+
[pyenv](https://github.com/pyenv/pyenv) can help (as can enabling Travis CI on
168+
your fork).
169+
170+
Due to its dynamic nature, you may run into false positives. In this case, you
171+
can add to the whitelists for each affected Python version in
172+
`tests/stubtest_whitelists`. Please file issues for stubtest false positives
173+
at [mypy](https://github.com/python/mypy/issues).
174+
175+
To run stubtest against third party stubs, it's easiest to use stubtest
176+
directly. stubtest can also help you find things missing from the stubs.
177+
178+
179+
### flake8
180+
181+
flake8 requires Python 3.6 or higher. Run using: `(.venv3)$ flake8`
182+
183+
Note typeshed uses the `flake8-pyi` and `flake8-bugbear` plugins.

‎tests/stubtest_test.py‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
#!/usr/bin/env python3
2+
"""Test typeshed using stubtest
3+
4+
stubtest is a script in the mypy project that compares stubs to the actual objects at runtime.
5+
Note that therefore the output of stubtest depends on which Python version it is run with.
6+
In typeshed CI, we run stubtest with each Python minor version from 3.5 through 3.8 inclusive.
7+
8+
We pin the version of mypy / stubtest we use in .travis.yml so changes to those don't break
9+
typeshed CI.
10+
11+
"""
12+
13+
from pathlib import Path
14+
import subprocess
15+
import sys
16+
17+
18+
def run_stubtest(typeshed_dir: Path) -> int:
19+
whitelist_dir = typeshed_dir / "tests" / "stubtest_whitelists"
20+
version_whitelist = "py{}{}.txt".format(sys.version_info.major, sys.version_info.minor)
21+
22+
cmd = [
23+
sys.executable,
24+
"-m",
25+
"mypy.stubtest",
26+
# Use --ignore-missing-stub, because if someone makes a correct addition, they'll need to
27+
# also make a whitelist change and if someone makes an incorrect addition, they'll run into
28+
# false negatives.
29+
"--ignore-missing-stub",
30+
"--check-typeshed",
31+
"--custom-typeshed-dir",
32+
str(typeshed_dir),
33+
"--whitelist",
34+
str(whitelist_dir / "py3_common.txt"),
35+
"--whitelist",
36+
str(whitelist_dir / version_whitelist),
37+
]
38+
if sys.version_info < (3, 8):
39+
# As discussed in https://github.com/python/typeshed/issues/3693, we only aim for
40+
# positional-only arg accuracy for the latest Python version.
41+
cmd += ["--ignore-positional-only"]
42+
try:
43+
print(" ".join(cmd), file=sys.stderr)
44+
subprocess.run(cmd, check=True)
45+
except subprocess.CalledProcessError as e:
46+
print(
47+
"\nNB: stubtest output depends on the Python version it is run with. See README.md "
48+
"for more details.\n"
49+
"NB: We only check positional-only arg accuracy for Python 3.8.\n"
50+
"If stubtest is complaining about 'unused whitelist entry' after your fix, please "
51+
"remove the entry from the whitelist file. Note you may have to do this for other "
52+
"version-specific whitelists as well. Thanks for helping burn the backlog of errors!\n"
53+
"\nCommand run was: {}\n".format(" ".join(cmd)),
54+
file=sys.stderr,
55+
)
56+
print("stubtest failed", file=sys.stderr)
57+
return e.returncode
58+
else:
59+
print("stubtest succeeded", file=sys.stderr)
60+
return 0
61+
62+
63+
if __name__ == "__main__":
64+
sys.exit(run_stubtest(typeshed_dir=Path(".")))

‎tests/stubtest_whitelists/py35.txt‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
asyncio.Future._callbacks
2+
asyncio.Future._exception
3+
asyncio.Future._loop
4+
asyncio.Future._tb_logger
5+
asyncio.Task.__init__
6+
asyncio.constants.DEBUG_STACK_DEPTH
7+
asyncio.constants.SENDFILE_FALLBACK_READBUFFER_SIZE
8+
asyncio.constants.SSL_HANDSHAKE_TIMEOUT
9+
asyncio.exceptions
10+
asyncio.futures.Future._callbacks
11+
asyncio.futures.Future._exception
12+
asyncio.futures.Future._loop
13+
asyncio.futures.Future._tb_logger
14+
asyncio.futures._TracebackLogger.__init__
15+
asyncio.protocols.BufferedProtocol
16+
asyncio.runners
17+
asyncio.tasks.Task.__init__
18+
bdb.GENERATOR_AND_COROUTINE_FLAGS
19+
builtins.str.maketrans
20+
cgi.escape
21+
cmath.log
22+
codecs.StreamRecoder.seek
23+
collections.Reversible
24+
collections.UserString.maketrans
25+
ctypes.CDLL.__init__
26+
decimal.Decimal.as_integer_ratio
27+
gettext.NullTranslations.npgettext
28+
gettext.NullTranslations.pgettext
29+
gettext.dnpgettext
30+
gettext.dpgettext
31+
gettext.npgettext
32+
gettext.pgettext
33+
importlib.metadata
34+
importlib.resources
35+
io.BufferedRandom.read1
36+
io.BufferedReader.read1
37+
io.StringIO.readline
38+
ipaddress._BaseNetwork.__init__
39+
macpath.basename
40+
macpath.commonpath
41+
macpath.commonprefix
42+
macpath.dirname
43+
macpath.getatime
44+
macpath.getctime
45+
macpath.getmtime
46+
macpath.getsize
47+
macpath.isabs
48+
macpath.isdir
49+
macpath.islink
50+
macpath.ismount
51+
macpath.join
52+
macpath.normpath
53+
macpath.realpath
54+
macpath.relpath
55+
macpath.samefile
56+
macpath.samestat
57+
macpath.split
58+
macpath.splitdrive
59+
macpath.splitext
60+
mmap.ACCESS_DEFAULT
61+
modulefinder.ModuleFinder.scan_opcodes
62+
multiprocessing.context.BaseContext.reducer
63+
multiprocessing.shared_memory
64+
os.DirEntry
65+
os.utime
66+
pyexpat.XMLParserType.ExternalEntityParserCreate
67+
smtpd.SMTPChannel.__init__
68+
smtpd.SMTPServer.__init__
69+
socket.NETLINK_CRYPTO
70+
sre_compile.dis
71+
sre_parse.GLOBAL_FLAGS
72+
sre_parse.Tokenizer.pos
73+
sre_parse.Verbose
74+
subprocess.check_output
75+
tracemalloc.Filter.__init__
76+
typing.AbstractSet.isdisjoint
77+
typing.Coroutine.cr_await
78+
typing.Coroutine.cr_code
79+
typing.Coroutine.cr_frame
80+
typing.Coroutine.cr_running
81+
typing.Generator.gi_code
82+
typing.Generator.gi_frame
83+
typing.Generator.gi_running
84+
typing.Generator.gi_yieldfrom
85+
typing.Mapping.get
86+
typing.MutableMapping.pop
87+
typing.MutableMapping.setdefault
88+
typing.MutableSequence.append
89+
typing.MutableSequence.extend
90+
typing.MutableSequence.insert
91+
typing.MutableSequence.remove
92+
typing.MutableSet.add
93+
typing.MutableSet.discard
94+
typing.MutableSet.remove
95+
typing.Protocol
96+
typing.Sequence.count
97+
typing.Sequence.index
98+
typing.Type
99+
typing.runtime_checkable
100+
unittest.async_case
101+
uuid.UUID.int
102+
uuid.getnode
103+
xml.etree.cElementTree.XMLPullParser
104+
xml.parsers.expat.XMLParserType.ExternalEntityParserCreate
105+
zipfile.ZipFile.open
106+
zlib.compress

0 commit comments

Comments
 (0)