`sage --fixdoctests --update-known-test-failures`; silence modularized distributions in CI #36264

mkoeppe · 2023-09-13T21:01:26Z

The distributions sagemath-repl and sagemath-categories run parts of the testsuite in a virtual environment created by tox when SAGE_CHECK=yes. This also runs as part of the Build & Test CI.

Here we make several improvements:

Support SAGE_CHECK=warn
Invoke the doctester with --baseline-stats-path={toxinidir}/known-test-failures.json
Make the tests silent unless new test failures, not recorded in the known-test-failures.json, are encountered

To help maintain the known-test-failures.json, we also add new features to the command sage --fixdoctests.

sage --fixdoctests --update-known-test-failures reads the stats files generated by the doctester in the virtual environments and writes updated known-test-failures.json files to the source tree
doctester stats now also include a field ntests - number of doctests of a module that were run

sage --fixdoctests also receives new switches --distribution all, --fixed-point, --verbose, --no-diff and some other improvements.

📝 Checklist

The title is concise, informative, and self-explanatory.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

Depends on Fix setuptools version bounds in generated pyproject.toml files #36263 (merged here)
Depends on sage -t: Distinguish .pxd from .pyx in doctest basenames #36238 (merged here)

… fails

kwankyu · 2023-09-19T12:23:01Z

pkgs/sagemath-environment/tox.ini

 package_env = .pkg-sagepython
+
+
+[testenv:sagepython-sagewheels-nopypi-norequirements]


single empty line

This break is intended - this sets apart the default environment

kwankyu · 2023-09-19T12:24:30Z

pkgs/sagemath-objects/tox.ini

 package_env = .pkg-sagepython
+
+
+[testenv:sagepython-sagewheels-nopypi-norequirements]


Two lines, intended?

Yes, this separates the default environment from the others above

kwankyu · 2023-09-19T12:28:05Z

pkgs/sagemath-repl/known-test-failures.json

+    },
+    "sage.repl.attach": {
+        "ntests": 128
+    },


No "failed: false"?

Or "success: true"?

No, the timings2.json files created by the doctester don't have these.

kwankyu · 2023-09-19T12:31:51Z

pkgs/sagemath-standard/tox.ini

+    # Build dependencies according to requirements.txt (all versions fixed).
+    # Use ONLY the wheels built and stored by the Sage distribution (no PyPI):
+    #
+    #   ./sage -sh -c '(cd pkgs/sagemath-standard && tox -v -v -v -e sagepython-sagewheels-nopypi)'


No sagepython-sagewheels-nopypi here?

No, the envlist only lists the default environments, i.e., those that will be tested if one uses tox without the -e switch.

OK. Thanks.

kwankyu · 2023-09-19T12:32:23Z

pkgs/sagemath-standard/tox.ini

-    #   ./sage -sh -c '(cd pkgs/sagemath-standard && tox -v -v -v -e sagepython-sagewheels-nopypi)'
-    #
-    sagepython-sagewheels-nopypi,
-    #


# # SUPPORTED ENVIRONMENTS: #

No?

I've changed the format from "SUPPORTED" / "OTHER SUPPORTED" to (unmarked) default / "OTHER SUPPORTED"

kwankyu · 2023-09-20T04:22:11Z

I got

$ sage --fixdoctests
sage-fixdoctests: At least one filename is required when --update-known-test-failures is not used
Running "sage -t -p "

Is it really running "sage -t -p "?

kwankyu · 2023-09-20T04:26:15Z

 $ sage --fixdoctests --help
...
  --probe FEATURES      check whether '# optional/needs' tags are still
                        needed, remove these
...

remove those not needed?

kwankyu · 2023-09-20T04:29:24Z

From

$ sage --fixdoctests --update-known-test-failures --verbose
Running "sage -t -p "

I don't understand the output. It seems not running anything.

kwankyu · 2023-09-20T05:36:13Z

"--fixed-point" sounds good by itself. But since it follows "--fixdoctests", it makes me stop and think. An alternative is "--stabilize". Just a suggestion.

kwankyu · 2023-09-20T05:58:17Z

$ sage --fixdoctests --distribution sagemath-objects --verbose src/sage/rings/function_field
Running "pkgs/sagemath-objects/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage -t --environment sage.all__sagemath_objects ./src/sage/version.py"
sh: pkgs/sagemath-objects/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage: No such file or directory
sage-fixdoctests: Doctester exited with error status 127

Is this normal? I ran this after make wheels.

kwankyu · 2023-09-20T06:02:21Z

$ sage --fixdoctests --distribution sagemath-categories --verbose src/sage/rings/function_field --fixed-point
##############################################################################
sage-fixdoctests: Fixing with --distribution=sagemath-categories
Running "pkgs/sagemath-categories/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage -t --environment sage.all__sagemath_categories ./src/sage/version.py"
Running "pkgs/sagemath-categories/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage -t -p --environment sage.all__sagemath_categories --if-installed src/sage/rings/function_field/all.py src/sage/rings/function_field/constructor.py src/sage/rings/function_field/derivations.py src/sage/rings/function_field/derivations_polymod.py 
...
src/sage/rings/function_field/valuation_ring.py"
usage: sage -t [options] filenames
sage-runtests: error: unrecognized arguments: --if-installed
sage-fixdoctests: Processing unprocessed files
sage-fixdoctests: No fixes made in 'src/sage/rings/function_field/place.py'
...
sage-fixdoctests: Fixed point reached

There is an error sage-runtests: error: unrecognized arguments: --if-installed above.

kwankyu · 2023-09-20T06:08:15Z

From

$ sage --fixdoctests --distribution sagemath-categories --verbose src/sage/rings/function_field/function_field.py --fixed-point
##############################################################################
sage-fixdoctests: Fixing with --distribution=sagemath-categories
Running "pkgs/sagemath-categories/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage -t --environment sage.all__sagemath_categories ./src/sage/version.py"
Running "pkgs/sagemath-categories/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage -t -p --environment sage.all__sagemath_categories --if-installed src/sage/rings/function_field/function_field.py"
usage: sage -t [options] filenames
sage-runtests: error: unrecognized arguments: --if-installed
sage-fixdoctests: Processing unprocessed files
sage-fixdoctests: No fixes made in 'src/sage/rings/function_field/function_field.py'
sage-fixdoctests: Fixed point reached

why does this run?

Running "pkgs/sagemath-categories/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage -t 
  --environment sage.all__sagemath_categories ./src/sage/version.py"

It seems that this runs if --distribution is given.

mkoeppe · 2023-09-20T17:18:24Z

$ sage --fixdoctests
sage-fixdoctests: At least one filename is required when --update-known-test-failures is not used
Running "sage -t -p "

Is it really running "sage -t -p "?

Thanks, fixed in 93f6a05

mkoeppe · 2023-09-20T17:25:09Z

$ sage --fixdoctests --help
...
  --probe FEATURES      check whether '# optional/needs' tags are still
                        needed, remove these
...

remove those not needed?

Thanks, done in e380f9a

mkoeppe · 2023-09-20T17:26:13Z

$ sage --fixdoctests --update-known-test-failures --verbose
Running "sage -t -p "
I don't understand the output. It seems not running anything.

It gives an error now after a61cdbf

mkoeppe · 2023-09-20T17:27:46Z

$ sage --fixdoctests --distribution sagemath-objects --verbose src/sage/rings/function_field
Running "pkgs/sagemath-objects/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage -t --environment sage.all__sagemath_objects ./src/sage/version.py"
sh: pkgs/sagemath-objects/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage: No such file or directory
sage-fixdoctests: Doctester exited with error status 127

Is this normal? I ran this after make wheels.

The .tox environments are only created if you run that with SAGE_CHECK=yes (or SAGE_CHECK=warn)

mkoeppe · 2023-09-20T17:28:25Z

There is an error sage-runtests: error: unrecognized arguments: --if-installed above.

This must have been an old .tox environment

mkoeppe · 2023-09-20T17:30:02Z

why does this run?

Running "pkgs/sagemath-categories/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage -t 
  --environment sage.all__sagemath_categories ./src/sage/version.py"

It seems that this runs if --distribution is given.

Before it tests the given files, it uses this command to see if the .tox environment is working. It skips those that do not seem to be working

mkoeppe · 2023-09-20T17:36:07Z

"--fixed-point" sounds good by itself. But since it follows "--fixdoctests", it makes me stop and think. An alternative is "--stabilize". Just a suggestion.

I also considered --until-stable but ultimately I thought that "stable" might be taken as promising something good, whereas "fixed point" expresses that it has converged to "something", which is closer to the truth

kwankyu · 2023-09-21T00:45:32Z

why does this run?
Running "pkgs/sagemath-categories/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage -t 
  --environment sage.all__sagemath_categories ./src/sage/version.py"
It seems that this runs if --distribution is given.
Before it tests the given files, it uses this command to see if the .tox environment is working. It skips those that do not seem to be working

Then how about not logging the line "Running ... ./src/sage/version.py", but instead logging the result of the check? That log looks internal.

kwankyu · 2023-09-21T00:52:50Z

$ sage --fixdoctests --distribution sagemath-objects --verbose src/sage/rings/function_field
Running "pkgs/sagemath-objects/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage -t --environment 
Is this normal? I ran this after `make wheels`.

The .tox environments are only created if you run that with SAGE_CHECK=yes (or SAGE_CHECK=warn)

The .tox environment is there, but sage is not in bin.

I tested again after SAGE_CHECK=yes make sagemath_objects. I get the same.

This

$sage --fixdoctests --distribution sagemath_categories --verbose src/sage/rings/function_field --fixed-point

works though. So the problem is with sagemath_objects.

mkoeppe · 2023-09-21T01:23:43Z

So the problem is with sagemath_objects

Ah, yes. The doctester is provided by sagemath-repl, which depends on sagemath-objects. Because of this, sagemath-objects is not tested using the doctester. (Also, the sage script is provided by sagemath-environment, which is also not installed in the tox environment of sagemath-objects.)

kwankyu · 2023-09-21T01:39:23Z

I suspected that sort of things. Then how about detecting such a situation, and warn the user of --fixdoctests. If --distribution all is used, then sagemath-objects should be skipped anyway instead of erroring out. Right?

mkoeppe · 2023-09-21T01:39:56Z

Before it tests the given files, it uses this command to see if the .tox environment is working. It skips those that do not seem to be working

Then how about not logging the line "Running ... ./src/sage/version.py", but instead logging the result of the check? That log looks internal.

I've changed the wording in e3c4b15; better?

mkoeppe · 2023-09-21T01:41:45Z

If --distribution all is used, then sagemath-objects should be skipped anyway

That's right, in #35095 'all' also expands to a list that does not include sagemath-objects.

kwankyu · 2023-09-21T01:48:50Z

$ sage --fixdoctests --distribution all src/sage/rings/function_field/function_field.py --fixed-point
##############################################################################
sage-fixdoctests: Fixing with --distribution=sagemath-categories
Running "pkgs/sagemath-categories/.tox/sagepython-sagewheels-nopypi-norequirements/bin/sage -t -p --environment sage.all__sagemath_categories --if-installed src/sage/rings/function_field/function_field.py"
sage-runtests: Skipping 'src/sage/rings/function_field/function_field.py' because module sage.rings.function_field cannot be imported
##############################################################################
sage-fixdoctests: Fixing with --distribution=''
Running "sage -t -p src/sage/rings/function_field/function_field.py"
sage-fixdoctests: Processing unprocessed files
sage-fixdoctests: No fixes made in 'src/sage/rings/function_field/function_field.py'
sage-fixdoctests: Fixed point reached

Why only the "empty distributon" and sagemath-categories?

kwankyu · 2023-09-21T01:50:00Z

Is the list of distributions hard-wired?

mkoeppe · 2023-09-21T01:50:48Z

Yes, it's hardwired; these are the only distributions for which the development branch has a useful testsuite.

Much more in #35095

…s to 'all'

…with 0 tests, no errors

…n-test-failures.json

…s on success

… simplify spkg-install

…ut only run tests if SAGE_CHECK

…seline'

… 10.2.beta2

…tion all (for 10.2.beta2)

…options

…s required'

…ithout distributions

…ilures

github-actions · 2023-09-22T01:52:06Z

Documentation preview for this PR (built with commit 23e9923; changes) is ready! 🎉

kwankyu

LGTM.

mkoeppe · 2023-09-22T16:26:10Z

Thank you!

Matthias Koeppe added 2 commits September 16, 2023 11:31

src/sage/doctest/control.py: Show details when loading baseline-stats…

fbf7cf3

… fails

src/sage/doctest/control.py: Pretty-print the timings file

da4e1fa

kwankyu reviewed Sep 19, 2023

View reviewed changes

Matthias Koeppe added 23 commits September 21, 2023 15:58

sage -fixdoctests --update-known-test-failures: Add more distribution…

e80cc09

…s to 'all'

src/bin/sage-fixdoctests: Add more distros for --update...

9aaa6fe

sage --fixdoctests --update-known-test-failures: Do not record files …

9c7499b

…with 0 tests, no errors

pkgs/sagemath-{repl,categories}: Test with --baseline-stats-path=know…

abab827

…n-test-failures.json

Use TOX_PARALLEL_NO_SPINNER=1

0c4c84f

build/pkgs/sagemath_objects/spkg-install: Print locations of log file…

6472853

…s on success

pkgs/*/tox.ini: Set envlist to what should be tested by spkg-install;…

9eacf0c

… simplify spkg-install

build/pkgs/sagemath_objects/spkg-install: Always create the toxenv, b…

5b820f2

…ut only run tests if SAGE_CHECK

pkgs/sagemath-categories/tox.ini: Add --probe all

ec6b67f

build/pkgs/sagemath*/spkg-install: Redirect output from sdh_store_wheel

d5dc4e9

build/pkgs/sagemath_objects/spkg-install: Rephrase messages using 'ba…

3a7b829

…seline'

pkgs/sagemath-{categories,repl}, src/bin/sage-fixdoctests: Update for…

4a81f74

… 10.2.beta2

./sage -fixdoctests --no-test --update-known-test-failures --distribu…

adf00c0

…tion all (for 10.2.beta2)

Remove file committed by accident

b0c8fbc

src/doc/en/developer/doctesting.rst: Update

2080dac

src/doc/en/developer/doctesting.rst: Document more sage --fixdoctest …

04a82b7

…options

sage --fixdoctests: Exit after error message 'At least one filename i…

d0e48de

…s required'

sage --fixdoctests: Improve help string

2a58ae9

sage --fixdoctests: Exit with error on --update-known-test-failures w…

0ff47b2

…ithout distributions

sage --fixdoctests: Better messages when testing the doctester

2f5dd2b

src/doc/en/developer/doctesting.rst: Fix grammar

fbc5df0

src/doc/en/developer/doctesting.rst: Expand on --update-known-test-fa…

f495618

…ilures

src/sage/doctest/reporting.py: Update doctest output

23e9923

kwankyu approved these changes Sep 22, 2023

View reviewed changes

kwankyu added s: positive review and removed s: needs review labels Sep 22, 2023

vbraun merged commit 342e32b into sagemath:develop Sep 24, 2023

github-actions bot removed the s: positive review label Sep 24, 2023

		package_env = .pkg-sagepython


		[testenv:sagepython-sagewheels-nopypi-norequirements]

Uh oh!

Uh oh!

sage --fixdoctests --update-known-test-failures; silence modularized distributions in CI #36264

sage --fixdoctests --update-known-test-failures; silence modularized distributions in CI #36264

Uh oh!

Conversation

mkoeppe commented Sep 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Checklist

⌛ Dependencies

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kwankyu commented Sep 20, 2023

Uh oh!

kwankyu commented Sep 20, 2023

Uh oh!

kwankyu commented Sep 20, 2023

Uh oh!

kwankyu commented Sep 20, 2023

Uh oh!

kwankyu commented Sep 20, 2023

Uh oh!

kwankyu commented Sep 20, 2023

Uh oh!

kwankyu commented Sep 20, 2023

Uh oh!

mkoeppe commented Sep 20, 2023

Uh oh!

mkoeppe commented Sep 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mkoeppe commented Sep 20, 2023

Uh oh!

mkoeppe commented Sep 20, 2023

Uh oh!

mkoeppe commented Sep 20, 2023

Uh oh!

mkoeppe commented Sep 20, 2023

Uh oh!

mkoeppe commented Sep 20, 2023

Uh oh!

kwankyu commented Sep 21, 2023

Uh oh!

kwankyu commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mkoeppe commented Sep 21, 2023

Uh oh!

kwankyu commented Sep 21, 2023

Uh oh!

mkoeppe commented Sep 21, 2023

Uh oh!

mkoeppe commented Sep 21, 2023

Uh oh!

kwankyu commented Sep 21, 2023

`sage --fixdoctests --update-known-test-failures`; silence modularized distributions in CI #36264

`sage --fixdoctests --update-known-test-failures`; silence modularized distributions in CI #36264

mkoeppe commented Sep 13, 2023 •

edited

Loading

mkoeppe commented Sep 20, 2023 •

edited

Loading

kwankyu commented Sep 21, 2023 •

edited

Loading