Add configure option --with-sage-venv=SAGE_VENV to create venv there instead of in SAGE_LOCAL

[mkoeppe]

#27824 made $SAGE_LOCAL a venv if a suitable system python3 is found.

In this ticket, we add support for installing builds of Sage with several python versions, sharing the non-Python packages in SAGE_LOCAL:

• keeping install records for Python packages (recognized by having install-requires.txt or requirements.txt) in $SAGE_VENV/var/lib/sage/{installed,scripts,wheels}, separate from those of non-Python packages ($SAGE_LOCAL/var/lib/sage/{installed,scripts})
• likewise for $SAGE_LOCAL/var/tmp/sage/build
• here $SAGE_VENV defaults to $SAGE_LOCAL, but can be overridden to an arbitrary directory that will be used as the wheel-building venv, for example $SAGE_LOCAL/var/lib/sage/venv/$PYTHON_TAG.

Configuration:

• This is activated by running, for example,
  ./configure --with-python=/usr/bin/python3.8 --with-sage-venv="$SAGE_LOCAL/var/lib/sage/venv/python3.8".

As of this ticket, we are able to establish the venv in an arbitrary configured directory instead of SAGE_LOCAL. This is already useful for #31396 to create a wheel-building venv that will not be packaged as part of SAGE_LOCAL.

In follow-up tickets, we gain the full functionality that facilitates testing with several Python versions without having to rebuild the Sage distribution. This depends on:

• Repackage pynac as a pip-installable package #30534 Repackage pynac as a pip-installable package

Follow-ups:

• We also want to support ./configure --with-sage-venv=no which would suppress making any venv (./configure --with-sage-venv=none #30896)

• Support the tricky case: When system python3 is not in use and SAGE_VENV != SAGE_LOCAL, then both a real $SAGE_LOCAL/bin/python3 needs to be built and a venv in SAGE_VENV created... or should python3 be installed in $SAGE_VENV?)

• Optional: A file build/pkgs/SPKG/trees could override the install tree determination - this would allow us to for example install a package such as jupyter_core both into something like SAGE_NOTEBOOK_VENV and SAGE_VENV...

CC: @dimpase @jdemeyer @embray @vbraun @jhpalmieri

Component: build

Keywords: sd111

Author: Matthias Koeppe

Branch/Commit: d68e861

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/29013

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-We propose to create a venv (https://docs.python.org/3/library/venv.html) at `$SAGE_LOCAL/venv`, 
+We propose to create a venv (https://docs.python.org/3/library/venv.html) at `$SAGE_LOCAL/lib/sage/venv/sage/`, 
 to activate the venv in the `sage-env` script, and to install all Python packages (including `sagelib`) into this venv.
 
 This is preparation for #27824, in which the system's python3 will be used when available.

[mkoeppe]

Branch: public/29013-use-py3-venv

[mkoeppe]

comment:3

A simple change. Compiles from a fresh checkout and runs without problems on macOS.

---

New commits:

1363425	In a python3 build, install all Python packages into a venv
a394268	sage-pip-install: Use PYTHON=sage-python23

[mkoeppe]

Commit: a394268

[mkoeppe]

comment:4

In the branch #27824, the venv is created by slightly more complicated code in build/make/Makefile. This complexity is not necessary for the present ticket.

[dimpase]

comment:5

Does this ticket make sense separately, or only as part of #27824 ?

[mkoeppe]

comment:6

#27824 is the use case that I have in mind.
This ticket is a less intrusive change and should go through the betas first.

[mkoeppe]

Dependencies: #29002

[jhpalmieri]

comment:8

I tried building with this branch on OS X, and I got doctest failures:

sage -t --long src/sage/plot/plot.py  # 1 doctest failed
sage -t --long src/sage/matrix/matrix2.pyx  # 2 doctests failed
sage -t --long src/sage/finance/time_series.pyx  # 5 doctests failed
sage -t --long src/sage/misc/trace.py  # 4 doctests failed
sage -t --long src/sage/env.py  # 1 doctest failed
sage -t --long src/sage/repl/configuration.py  # 1 doctest failed
sage -t --long src/sage/schemes/affine/affine_homset.py  # 1 doctest failed

For example

File "src/sage/plot/plot.py", line 521, in sage.plot.plot
Failed example:
    os.system("sage -c \"if 'matplotlib' in sys.modules: sys.exit(1)\"") # long time
Expected:
    0
Got:
    Traceback (most recent call last):
      File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.1.beta0/local/bin/sage-eval", line 4, in <module>
        from sage.all import *
    ModuleNotFoundError: No module named 'sage'
    256

and

**********************************************************************
File "src/sage/matrix/matrix2.pyx", line 1127, in sage.matrix.matrix2.Matrix.pseudoinverse
Failed example:
    M.pseudoinverse()  # tol 1e-15
Expected:
    [0.0620518477661335 0.0206839492553778 0.0124103695532267]
    [ 0.124103695532267 0.0413678985107557 0.0248207391064534]
    [ 0.186155543298400 0.0620518477661335 0.0372311086596801]
Got:
    [0.0620518477661334 0.0206839492553778 0.0124103695532267]
    [ 0.124103695532267 0.0413678985107556 0.0248207391064534]
    [ 0.186155543298400 0.0620518477661335 0.0372311086596801]
Tolerance exceeded in 2 of 9:
    0.0620518477661335 vs 0.0620518477661334, tolerance 2e-15 > 1e-15
    0.0413678985107557 vs 0.0413678985107556, tolerance 3e-15 > 1e-15
**********************************************************************
File "src/sage/matrix/matrix2.pyx", line 1131, in sage.matrix.matrix2.Matrix.pseudoinverse
Failed example:
    M.pseudoinverse(algorithm="numpy")  # tol 1e-15
Expected:
    [0.0620518477661335 0.0206839492553778 0.0124103695532267]
    [ 0.124103695532267 0.0413678985107557 0.0248207391064534]
    [ 0.186155543298400 0.0620518477661335 0.0372311086596801]
Got:
    [0.0620518477661334 0.0206839492553778 0.0124103695532267]
    [ 0.124103695532267 0.0413678985107556 0.0248207391064534]
    [ 0.186155543298400 0.0620518477661335 0.0372311086596801]
Tolerance exceeded in 2 of 9:
    0.0620518477661335 vs 0.0620518477661334, tolerance 2e-15 > 1e-15
    0.0413678985107557 vs 0.0413678985107556, tolerance 3e-15 > 1e-15
**********************************************************************
1 item had failures:
   2 of  25 in sage.matrix.matrix2.Matrix.pseudoinverse
    [2472 tests, 2 failures, 9.06 s]

[mkoeppe]

comment:9

Thanks for testing!

[mkoeppe]

Changed dependencies from #29002 to #29002, #29022

[mkoeppe]

comment:10

Replying to @jhpalmieri:

sage -t --long src/sage/matrix/matrix2.pyx # 2 doctests failed

Not sure what's causing the numerical failures.

sage -t --long src/sage/env.py # 1 doctest failed

For this one, I have created #29022.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

d4fbe40

_get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL

[sagetrac-git]

Changed commit from a394268 to d4fbe40