Create PEP 503 simple repository for wheels built during installation

[mkoeppe]

As a follow-up to #29500 (Install all Python packages via pip wheel, store wheels in $SAGE_LOCAL/var/lib/sage/wheels):

We create a PEP 503 compliant Simple Repository index in $SAGE_SPKG_WHEELS. Then users can create their virtual environment using tools such as pipenv [[source]] or pip install --index-url.

We do this using the command dir2pi from https://pypi.org/project/pip2pi/

Follow-up:

• Add src/requirements.txt for installation of sagelib in a venv #30578 - Add src/requirements.txt for installation of sagelib in a venv

---

References:

• Implement pep 503 Simple Repository API for deployment pytorch/pytorch#25639 "Implement pep 503 Simple Repository API for deployment"

• https://pypi.org/project/piprepo/ - create PEP 503 package repo

• https://stackoverflow.com/questions/45744937/how-do-install-packages-from-a-local-python-package-index

• https://pipenv-fork.readthedocs.io/en/latest/advanced.html#specifying-package-indexes

CC: @slel @tobiasdiez @embray @jhpalmieri

Component: build

Keywords: sd111

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/pep-503-simple-repository-for-wheels @ 333e50c

Reviewer: Tobias Diez

Issue created by migration from https://trac.sagemath.org/ticket/30527

[mkoeppe]

comment:1

piprepo can be used for this task, but it has excessive dependencies

[mkoeppe]

Dependencies: #29500

[tobiasdiez]

comment:3

There is also pypiserver, see eg https://stackoverflow.com/a/51528588/873661 for how to use it with pipenv

[mkoeppe]

comment:4

Well, we don't want to run a server for this, but following this link I found https://github.com/wolever/pip2pi

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,6 +1,6 @@
 As a follow-up to #29500 (Install all Python packages via `pip wheel`, store wheels in `$SAGE_LOCAL/var/lib/sage/wheels`):
 
-We create a PEP 503 compliant Simple Repository index in `$SAGE_SPKG_WHEELS`. Then users can create their virtual environment using tools such as `pipenv` `[[source]]` or `pip install --index-url`.
+We create a [PEP 503](https://www.python.org/dev/peps/pep-0503/) compliant Simple Repository index in `$SAGE_SPKG_WHEELS`. Then users can create their virtual environment using tools such as `pipenv` `[[source]]` or `pip install --index-url`.
 
 ---
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,6 +1,10 @@
 As a follow-up to #29500 (Install all Python packages via `pip wheel`, store wheels in `$SAGE_LOCAL/var/lib/sage/wheels`):
 
 We create a [PEP 503](https://www.python.org/dev/peps/pep-0503/) compliant Simple Repository index in `$SAGE_SPKG_WHEELS`. Then users can create their virtual environment using tools such as `pipenv` `[[source]]` or `pip install --index-url`.
+
+We do this using the command `dir2pi` from https://pypi.org/project/pip2pi/
+
+
 
 ---
 

[tobiasdiez]

comment:7

Yeah I agree, a server is not required (although the overhead of creating a simple server on listing on localhost should be pretty minimal). If pip2pi works with pipenv and pip, then that's perfect!

[mkoeppe]

Branch: u/mkoeppe/pep-503-simple-repository-for-wheels

[mkoeppe]

New commits:

8aa6fd9	build/bin/sage-dist-helpers (sdh_pip_install): Build a wheel, store it
d369aab	build/bin/sage-dist-helpers (sdh_store_and_pip_install_wheel): New, factored out from sdh_pip_install
2d435ab	build/pkgs/numpy/spkg-install.in: Install via setup.py bdist_wheel
55993b6	build/bin/sage-dist-helpers: Fixup
0a64674	build/pkgs/gambit/spkg-install.in: Install via bdist_wheel
ca58693	build/pkgs/pillow/spkg-install.in: Install via bdist_wheel
207d80f	build/pkgs/pip2pi: New
2555e19	build/make/install: At the end, update the repository index

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Commit: 2555e19

[tobiasdiez]

comment:10

What is the reason that sage needs to build wheels for standard packages, and not just downloads them from pypi?

Is it because some packages have patches applied? In this case, I would propose to migrate these patches to proper forks of the packages which are then published to pypi including their wheel. This has a few advantages, mainly that the build of sage is speedup and simplified, and that the wider python community can profit from these patches.

What do you think?

[mkoeppe]

comment:11

Replying to @tobiasdiez:

What is the reason that sage needs to build wheels for standard packages, and not just downloads them from pypi?

Sage-the-distribution is a deployment mechanism. Compiling all packages from source gives us full control and eliminates possible ABI problems from mixing prebuilt wheels that were compiled on different systems. Building from source also has advantages such as generating architecture-specific code (see #27122).

Is it because some packages have patches applied?

In some cases, yes, in other cases, just specific configuration.

In this case, I would propose to migrate these patches to proper forks of the packages which are then published to pypi including their wheel. This has a few advantages, mainly that the build of sage is speedup and simplified, and that the wider python community can profit from these patches.

We don't have intentions to take over maintenance of these packages. Whenever possible, we send our patches upstream. In many cases, it is just a matter of mismatched release schedules, and we have to keep patches only for a limited time until upstream makes a new stable release including a particular patch. For example, consider the numpy patch added in #29766. It is actually from upstream, but not included in any stable release. But we need the patch because we want Sage-the-distribution to support a particular platform.

[tobiasdiez]

comment:12

Thanks for the detailed explanation. Makes things clearer for me now.

Compiling all packages from source gives us full control and eliminates possible ABI problems from mixing prebuilt wheels that were compiled on different systems. Building from source also has advantages such as generating architecture-specific code

But isn't the concept of compile-once-reuse-often the main point of wheels? By design, it's the responsibility of the the maintainer (= publisher of the wheel) that the wheel is useable and optimized for the particular platforms they are published. This also make sense since she is the export of the code of the library.

I feel none of the advantages of wheels outlined in https://www.python.org/dev/peps/pep-0427/ can really prove effective if wheels are rebuilt from code every time.

The wheel binary package format frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment.

In my opinion, the build process of sage would immensely profit from all these advantages.

Anyway, this discussion is somewhat orthogonal to this ticket (sorry!). So concerning the changes, the code looks good to me. I'm only wondering if the pip2 support in the uninstallation script is really necessary (since only python3 is supported now). However, I don't feel familiar enough with the build script to change the status to positive review.

[tobiasdiez]

Reviewer: Tobias Diez,