We can now untokenize PEP 701 syntax, `test_tokenize` needs an update

[sobolevn]

Bug report

It states:

cpython/Lib/test/test_tokenize.py

Lines 1911 to 1912 in baefbb2

	# TODO: Remove this once we can untokenize PEP 701 syntax
	testfiles.remove(os.path.join(tempdir, "test_fstring.py"))

But, it works now 🎉
So, I guess we need to test f strings now, since it is an important part of Python's syntax.

Before:

» ./python.exe -m test test_tokenize
Using random seed 465266543
0:00:00 load avg: 2.63 Run 1 test sequentially
0:00:00 load avg: 2.63 [1/1] test_tokenize

== Tests result: SUCCESS ==

1 test OK.

Total duration: 857 ms
Total tests: run=107
Total test files: run=1/1
Result: SUCCESS

After:

» ./python.exe -m test test_tokenize
Using random seed 1341397923
0:00:00 load avg: 2.58 Run 1 test sequentially
0:00:00 load avg: 2.58 [1/1] test_tokenize

== Tests result: SUCCESS ==

1 test OK.

Total duration: 1.2 sec
Total tests: run=107
Total test files: run=1/1
Result: SUCCESS

Linked PRs

• gh-111031: Check more files in test_tokenize #111032
• [3.12] gh-111031: Check more files in test_tokenize (GH-111032) #111061

[sunmy2019]

All files except for test_fstring.py can be untokenized now.

[AlexWaygood]

As discussed in #111032 (review), it seems we can't, in fact, untokenize PEP-701 syntax just yet (but the linked PRs expanded the test files tested by test_tokenize nonetheless, which is great!).

Anything else to do here @sobolevn, or shall we close this out now? :)

[sobolevn]

Yes, let's close it :)