`sage.env`: Remove fallback from SAGE_SRC to SAGE_LIB #37666

mkoeppe · 2024-03-24T21:11:07Z

This is:

part of Meta-ticket: Remove use of SAGE_LIB and SAGE_EXTCODE variables #33037

📝 Checklist

The title is concise and informative.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

Depends on Fix doctest basenames when Sage library cannot be found with SAGE_LIB #37647
Depends on sage.doctest: Remove most uses of SAGE_SRC in doctests #37648
Depends on src/sage/features/sagemath.py: Add feature SAGE_SRC #37650

kiwifb · 2024-03-31T22:39:47Z

I haven''t thought too deeply about it so far. What will happen with doctests? I cannot remember if --installed make it look at SAGE_LIB instead of SAGE_SRC, it was too long ago.

antonio-rojas · 2024-03-31T22:43:48Z

This beaks (at least) doctesting

sage -t --long --random-seed=339726944106276385069493772491200419175 /usr/lib/python3.11/site-packages/sage/all.py
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/sage/doctest/reporting.py", line 415, in report
    basename = source.basename
               ^^^^^^^^^^^^^^^
  File "sage/misc/lazy_attribute.pyx", line 126, in sage.misc.lazy_attribute._lazy_attribute.__get__ (build/cythonized/sage/misc/lazy_attribute.c:3284)
    result = self.f(a)
  File "/usr/lib/python3.11/site-packages/sage/doctest/sources.py", line 687, in basename
    return get_basename(self.path)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sage/doctest/sources.py", line 102, in get_basename
    if path.startswith(dev):
       ^^^^^^^^^^^^^^^^^^^^
TypeError: startswith first arg must be str or a tuple of str, not NoneType

kiwifb · 2024-03-31T22:47:35Z

That's what I thought.

antonio-rojas · 2024-03-31T22:52:08Z

Easy fix for that issue:

diff --git a/src/sage/doctest/sources.py b/src/sage/doctest/sources.py
index fdad2749083..f8381f41af2 100644
--- a/src/sage/doctest/sources.py
+++ b/src/sage/doctest/sources.py
@@ -99,7 +99,7 @@ def get_basename(path):
     # the directory structure
     dev = SAGE_SRC
     sp = SAGE_LIB
-    if path.startswith(dev):
+    if dev and path.startswith(dev):
         # there will be a branch name
         i = path.find(os.path.sep, len(dev))
         if i == -1:

Running the full test suite now, will report tomorrow with other issues found.

mkoeppe · 2024-03-31T22:55:56Z

Right, this will depend on:

Fix doctest basenames when Sage library cannot be found with SAGE_LIB #37647

This lets you specify SAGE_SRC like /path/to/test/dir/../../src/dir, which avoids post-install test failures in sage/doctest/test.py

as it resolves symlinks

…gnize nodoctest marker files

antonio-rojas · 2024-04-01T07:14:34Z

Right, this will depend on:

Fix doctest basenames when Sage library cannot be found with SAGE_LIB #37647

It's broken with #37647 too

sage -t --long --random-seed=73818697356865316381657802103479065922 /usr/lib/python3.11/site-packages/sage/all.py
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/sage/doctest/reporting.py", line 415, in report
    basename = source.basename
               ^^^^^^^^^^^^^^^
  File "sage/misc/lazy_attribute.pyx", line 126, in sage.misc.lazy_attribute._lazy_attribute.__get__ (build/cythonized/sage/misc/lazy_attribute.c:3284)
    result = self.f(a)
  File "/usr/lib/python3.11/site-packages/sage/doctest/sources.py", line 704, in basename
    return get_basename(self.path)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sage/doctest/sources.py", line 112, in get_basename
    dev = os.path.realpath(SAGE_SRC)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 415, in realpath
TypeError: expected str, bytes or os.PathLike object, not NoneType

mkoeppe · 2024-04-01T07:58:56Z

Thanks for testing!

… new doctests

… is None

github-actions · 2024-04-01T18:53:46Z

Documentation preview for this PR (built with commit 82a32da; changes) is ready! 🎉

antonio-rojas · 2024-04-02T17:12:07Z

Still many uses of SAGE_SRC left causing failures

sage -t --long /usr/lib/python3.11/site-packages/sage/repl/rich_output/backend_doctest.py  # 14 doctests failed
sage -t --long /usr/lib/python3.11/site-packages/sage/repl/rich_output/display_manager.py  # 4 doctests failed
sage -t --long /usr/lib/python3.11/site-packages/sage/repl/rich_output/output_graphics3d.py  # 25 doctests failed
sage -t --long /usr/lib/python3.11/site-packages/sage/repl/rich_output/output_video.py  # 14 doctests failed
sage -t --long /usr/lib/python3.11/site-packages/sage/repl/rich_output/pretty_print.py  # 4 doctests failed
sage -t --long /usr/lib/python3.11/site-packages/sage/repl/rich_output/buffer.py  # 11 doctests failed
sage -t --long /usr/lib/python3.11/site-packages/sage/env.py  # 1 doctest failed
sage -t --long /usr/lib/python3.11/site-packages/sage/doctest/test.py  # 36 doctests failed
sage -t --long /usr/lib/python3.11/site-packages/sage/doctest/control.py  # 42 doctests failed
sage -t --long /usr/lib/python3.11/site-packages/sage/doctest/forker.py  # 42 doctests failed
sage -t --long /usr/lib/python3.11/site-packages/sage/doctest/reporting.py  # 1 doctest failed
sage -t --long /usr/lib/python3.11/site-packages/sage/doctest/sources.py  # 10 doctests failed
sage -t --long /usr/lib/python3.11/site-packages/sage/misc/edit_module.py  # 2 doctests failed
sage -t --long /usr/lib/python3.11/site-packages/sage/misc/sagedoc.py  # 22 doctests failed
sage -t --long /usr/lib/python3.11/site-packages/sage/tests/cmdline.py  # 2 doctests failed

Also, an error is thrown at the end of doctesting:

Traceback (most recent call last):
  File "/usr/bin/sage-runtests", line 172, in <module>
    if not all(os.path.isfile(os.path.join(SAGE_SRC, f))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/sage-runtests", line 172, in <genexpr>
    if not all(os.path.isfile(os.path.join(SAGE_SRC, f))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 76, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType

Finally, optional distributions fail to build:

Updating Cython code....
************************************************************************
Traceback (most recent call last):
  File "/build/sagemath-git/src/sage/pkgs/sagemath-bliss/setup.py", line 58, in <module>
    setup(
  File "/usr/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/usr/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
    super().run_command(command)
  File "/usr/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
    super().run_command(command)
  File "/usr/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/build/sagemath-git/src/sage/pkgs/sage-setup/sage_setup/command/sage_build_ext.py", line 25, in run
    self.run_command('build_cython')
  File "/usr/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
    super().run_command(command)
  File "/usr/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/build/sagemath-git/src/sage/pkgs/sage-setup/sage_setup/command/sage_build_cython.py", line 220, in run
    extensions = cythonize(
                 ^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1010, in cythonize
    module_list, module_metadata = create_extension_list(
                                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 886, in create_extension_list
    m, metadata = create_extension(template, kwds)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/sagemath-git/src/sage/pkgs/sage-setup/sage_setup/command/sage_build_cython.py", line 313, in create_extension
    return default_create_extension(template, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 762, in default_create_extension
    depends = resolve_depends(kwds['depends'], include_dirs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 469, in resolve_depends
    path = resolve_depend(depend, include_dirs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Cython/Utils.py", line 90, in Cython.Utils.cached_function.wrapper
  File "/usr/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 480, in resolve_depend
    path = join_path(dir, depend)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "Cython/Utils.py", line 90, in Cython.Utils.cached_function.wrapper
  File "<frozen posixpath>", line 76, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType
************************************************************************

Matthias Koeppe and others added 5 commits March 31, 2024 17:52

Make sage.doctest.sources.get_basename work in more situations

f1e9df7

This lets you specify SAGE_SRC like /path/to/test/dir/../../src/dir, which avoids post-install test failures in sage/doctest/test.py

Use realpath instead of normpath

03cb9d7

as it resolves symlinks

Silence pyflakes warnings

95e962d

src/sage/doctest/sources.py (get_basename): Do not use SAGE_LIB, reco…

a30659e

…gnize nodoctest marker files

src/sage/features/sagemath.py: Add feature SAGE_SRC

0aafe3f

Matthias Koeppe added 6 commits April 1, 2024 11:07

src/sage/env.py: Remove fallback SAGE_SRC -> SAGE_LIB

aa6c99c

Merge branch 'doctest_basename' into SAGE_SRC-no-fallback

b1de061

Merge branch 'doctest_remove_SAGE_SRC' into SAGE_SRC-no-fallback

72edd5c

Merge branch 'feature_SAGE_SRC' into SAGE_SRC-no-fallback

68cc101

src/sage/doctest/sources.py: Remove/conditionalize use of SAGE_SRC is…

08a6637

… new doctests

src/sage/doctest/sources.py (get_basename): Do not fail when SAGE_SRC…

82a32da

… is None

vbraun force-pushed the develop branch from eba5e19 to e5f42fa Compare June 3, 2024 22:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

`sage.env`: Remove fallback from SAGE_SRC to SAGE_LIB #37666

`sage.env`: Remove fallback from SAGE_SRC to SAGE_LIB #37666

Uh oh!

mkoeppe commented Mar 24, 2024 •

edited

Loading

Uh oh!

kiwifb commented Mar 31, 2024

Uh oh!

antonio-rojas commented Mar 31, 2024

Uh oh!

kiwifb commented Mar 31, 2024

Uh oh!

antonio-rojas commented Mar 31, 2024

Uh oh!

mkoeppe commented Mar 31, 2024

Uh oh!

antonio-rojas commented Apr 1, 2024

Uh oh!

mkoeppe commented Apr 1, 2024

Uh oh!

github-actions bot commented Apr 1, 2024

Uh oh!

antonio-rojas commented Apr 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

sage.env: Remove fallback from SAGE_SRC to SAGE_LIB #37666

Are you sure you want to change the base?

sage.env: Remove fallback from SAGE_SRC to SAGE_LIB #37666

Uh oh!

Conversation

mkoeppe commented Mar 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Checklist

⌛ Dependencies

Uh oh!

kiwifb commented Mar 31, 2024

Uh oh!

antonio-rojas commented Mar 31, 2024

Uh oh!

kiwifb commented Mar 31, 2024

Uh oh!

antonio-rojas commented Mar 31, 2024

Uh oh!

mkoeppe commented Mar 31, 2024

Uh oh!

antonio-rojas commented Apr 1, 2024

Uh oh!

mkoeppe commented Apr 1, 2024

Uh oh!

github-actions bot commented Apr 1, 2024

Uh oh!

antonio-rojas commented Apr 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

`sage.env`: Remove fallback from SAGE_SRC to SAGE_LIB #37666

`sage.env`: Remove fallback from SAGE_SRC to SAGE_LIB #37666

mkoeppe commented Mar 24, 2024 •

edited

Loading