From 06f16d2b477f348de9f8364e5d27430894207f46 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Mon, 26 Feb 2024 12:16:08 -0800 Subject: [PATCH 1/3] src/doc/en/developer/packaging.rst: Move section on patching to end, add subsection on importing patch from GitHub PR --- src/doc/en/developer/packaging.rst | 434 ++++++++++++++++------------- 1 file changed, 246 insertions(+), 188 deletions(-) diff --git a/src/doc/en/developer/packaging.rst b/src/doc/en/developer/packaging.rst index 3e014169a7c..14b83d6701c 100644 --- a/src/doc/en/developer/packaging.rst +++ b/src/doc/en/developer/packaging.rst @@ -75,6 +75,8 @@ the following source types: - its version number is defined by the required file ``package-version.txt``; + - can be patched; + - Sage installs the package using build and install scripts (see :ref:`section-spkg-install`); @@ -92,6 +94,8 @@ the following source types: - its version number is defined by the required file ``package-version.txt``; + - cannot be patched; + - no build and install scripts are needed (with one exception: the package :ref:`spkg_pip` installs itself from its wheel using a custom install script); @@ -109,6 +113,8 @@ the following source types: contains the name of the package (more details at https://pip.pypa.io/en/stable/user_guide/#requirements-files); + - cannot be patched; + - Sage installs the package using the ``pip`` package manager; - Sage delegates the recording of installed package version numbers to it; @@ -121,6 +127,8 @@ the following source types: - the file ``package-version.txt`` is optional; + - may be associated with a source tree included in the repository; + - installing the package runs the installation script ``spkg-install`` or ``spkg-install.in`` (see :ref:`section-spkg-install`); @@ -157,8 +165,7 @@ Third-party packages in Sage consist of two parts: the filename of the tarball. In any case, the actual code must be unmodified: if you need to change the sources, add a :ref:`patch ` - instead. See also :ref:`section-spkg-src` for automating the - modifications to the upstream tarball. + instead. #. The build scripts and associated files are in a subdirectory ``SAGE_ROOT/build/pkgs/``, where you replace ```` @@ -215,8 +222,8 @@ See :ref:`section-package-types` for the meaning of these types. .. _section-spkg-install: -Build and install scripts of normal packages --------------------------------------------- +Build and install scripts of ``normal`` packages +------------------------------------------------ The ``spkg-build.in`` and ``spkg-install.in`` files are templates for ``bash`` scripts ``spkg-build`` and ``spkg-install``, which build @@ -359,14 +366,27 @@ at build time, which should to the appropriate system-specific packages should prefix all commands in ``spkg-install.in`` that write into the installation hierarchy with ``$SAGE_SUDO``. -Install scripts of script packages ----------------------------------- +If an ``spkg-src`` file is present, it indicates that the tarball is not +an unmodified third-party tarball (see :ref:`section-spkg-patching`). +It documents how the tarball was generated (either by changing an upstream +tarball or generating it from an repository). As ideally +our tarballs are not modified, for most packages there is no ``spkg-src`` file. + -For script packages, it is also possible to use an install script named ``spkg-install``. +Install and source scripts of ``script`` packages +------------------------------------------------- + +For ``script`` packages, it is also possible to use an install script named ``spkg-install``. It needs to be an executable shell script; it is not subject to the templating described in the previous section and will be executed directly from the build directory. +Most of our ``script`` packages are associated with a source tree included in the +repository, in a subdirectory of ``$SAGE_ROOT/pkgs/``. In this case, there +is a symlink ``src`` that points to the source tree and a script ``spkg-src`` +that builds a tarball for the package. + + .. _section-sdh-helpers: Helper functions @@ -826,184 +846,6 @@ word ``SAGE_VENV``, and ``build/pkgs/sagemath_doc_html/trees.txt`` contains the word ``SAGE_DOCS``. -.. _section-spkg-patching: - -Patching sources ----------------- - -Actual changes to the source code must be via patches, which should be placed -in the ``patches/`` directory, and must have the ``.patch`` extension. GNU -patch is distributed with Sage, so you can rely on it being available. Patches -must include documentation in their header (before the first diff hunk), and -must have only one "prefix" level in the paths (that is, only one path level -above the root of the upstream sources being patched). So a typical patch file -should look like this: - -.. CODE-BLOCK:: diff - - Add autodoc_builtin_argspec config option - - Following the title line you can add a multi-line description of - what the patch does, where you got it from if you did not write it - yourself, if they are platform specific, if they should be pushed - upstream, etc... - - diff -dru Sphinx-1.2.2/sphinx/ext/autodoc.py.orig Sphinx-1.2.2/sphinx/ext/autodoc.py - --- Sphinx-1.2.2/sphinx/ext/autodoc.py.orig 2014-03-02 20:38:09.000000000 +1300 - +++ Sphinx-1.2.2/sphinx/ext/autodoc.py 2014-10-19 23:02:09.000000000 +1300 - @@ -1452,6 +1462,7 @@ - - app.add_config_value('autoclass_content', 'class', True) - app.add_config_value('autodoc_member_order', 'alphabetic', True) - + app.add_config_value('autodoc_builtin_argspec', None, True) - app.add_config_value('autodoc_default_flags', [], True) - app.add_config_value('autodoc_docstring_signature', True, True) - app.add_event('autodoc-process-docstring') - -Patches directly under the ``patches/`` directly are applied automatically -before running the ``spkg-install`` script (so long as they have the ``.patch`` -extension). If you need to apply patches conditionally (such as only on -a specifically platform), you can place those patches in a subdirectory of -``patches/`` and apply them manually using the ``sage-apply-patches`` script. -For example, considering the layout: - -.. CODE-BLOCK:: text - - SAGE_ROOT/build/pkgs/foo - |-- patches - | |-- solaris - | | |-- solaris.patch - | |-- bar.patch - | `-- baz.patch - -The patches ``bar.patch`` and ``baz.patch`` are applied to the unpacked -upstream sources in ``src/`` before running ``spkg-install``. To conditionally -apply the patch for Solaris the ``spkg-install`` should contain a section like -this: - -.. CODE-BLOCK:: bash - - if [ $UNAME == "SunOS" ]; then - sage-apply-patches -d solaris - fi - -where the ``-d`` flag applies all patches in the ``solaris/`` subdirectory of -the main ``patches/`` directory. - - -.. _section-spkg-patch-or-repackage: - -When to patch, when to repackage, when to autoconfiscate --------------------------------------------------------- - -- Use unpatched original upstream tarball when possible. - - Sometimes it may seem as if you need to patch a (hand-written) - ``Makefile`` because it "hard-codes" some paths or compiler flags: - - .. CODE-BLOCK:: diff - - --- a/Makefile - +++ b/Makefile - @@ -77,7 +77,7 @@ - # This is a Makefile. - # Handwritten. - - -DESTDIR = /usr/local - +DESTDIR = $(SAGE_ROOT)/local - BINDIR = $(DESTDIR)/bin - INCDIR = $(DESTDIR)/include - LIBDIR = $(DESTDIR)/lib - - Don't use patching for that. Makefile variables can be overridden - from the command-line. Just use the following in ``spkg-install``: - - .. CODE-BLOCK:: bash - - $(MAKE) DESTDIR="$SAGE_ROOT/local" - -- Check if Debian or another distribution already provides patches - for upstream. Use them, don't reinvent the wheel. - -- If the upstream Makefile does not build shared libraries, - don't bother trying to patch it. - - Autoconfiscate the package instead and use the standard facilities - of Automake and Libtool. This ensures that the shared library build - is portable between Linux and macOS. - -- If you have to make changes to ``configure.ac`` or other source - files of the autotools build system (or if you are autoconfiscating - the package), then you can't use patching; make a :ref:`modified - tarball ` instead. - -- If the patch would be huge, don't use patching. Make a - :ref:`modified tarball ` instead. - -- Otherwise, :ref:`maintain a set of patches - `. - - -.. _section-spkg-patch-maintenance: - -How to maintain a set of patches --------------------------------- - -We recommend the following workflow for maintaining a set of patches. - -- Fork the package and put it on a public git repository. - - If upstream has a public version control repository, import it from - there. If upstream does not have a public version control - repository, import the current sources from the upstream tarball. - Let's call the branch ``upstream``. - -- Create a branch for the changes necessary for Sage, let's call it - ``sage_package_VERSION``, where ``version`` is the upstream version - number. - -- Make the changes and commit them to the branch. - -- Generate the patches against the ``upstream`` branch: - - .. CODE-BLOCK:: bash - - rm -Rf SAGE_ROOT/build/pkgs/PACKAGE/patches - mkdir SAGE_ROOT/build/pkgs/PACKAGE/patches - git format-patch -o SAGE_ROOT/build/pkgs/PACKAGE/patches/ upstream - -- Optionally, create an ``spkg-src`` file in the Sage package's - directory that regenerates the patch directory using the above - commands. - -- When a new upstream version becomes available, merge (or import) it - into ``upstream``, then create a new branch and rebase it on top of - the updated upstream: - - .. CODE-BLOCK:: bash - - git checkout sage_package_OLDVERSION - git checkout -b sage_package_NEWVERSION - git rebase upstream - - Then regenerate the patches. - - -.. _section-spkg-src: - -Modified tarballs ------------------ - -The ``spkg-src`` file is optional and only to document how the upstream -tarball was changed. Ideally it is not modified, then there would be no -``spkg-src`` file present either. - -However, if you really must modify the upstream tarball then it is -recommended that you write a script, called ``spkg-src``, that makes the -changes. This not only serves as documentation but also makes it easier -to apply the same modifications to future versions. - - .. _section-spkg-versioning: Package versioning @@ -1023,9 +865,8 @@ This particular Sage package for ``database_stein_watkins`` was created in 2014, but the data it contains was last updated in 2011. If you apply any patches, or if you made changes to the upstream tarball -(see :ref:`section-directory-structure` for allowable changes), -then you should append a ``.p0`` to the version to indicate that it's -not a vanilla package. +(see :ref:`section-spkg-patching` below), then you should append a ``.p0`` +to the version to indicate that it's not an unmodified package. Additionally, whenever you make changes to a package *without* changing the upstream tarball (for example, you add an additional patch or you @@ -1366,3 +1207,220 @@ must meet the following requirements: :ref:`chapter-github`. +.. _section-spkg-patching: + +Modifying third-party code +========================== + +In the Sage distribution, we try to use unpatched original upstream tarballs +of stable versions of third-party packages whenever possible. + +Sometimes, however, modifications are necessary, either to fix a bug or +to make the package build on the platforms supported by Sage. + + +.. _section-spkg-patch-or-repackage: + +When to patch, when to repackage, when to autoconfiscate +-------------------------------------------------------- + +- First check whether there is already a newer stable version of the package + available that fixes the problem. In this case, try to upgrade the package. + +- Check if Debian or another distribution already provides patches + for upstream. Use them if possible, don't reinvent the wheel. + +- If the upstream project is maintained on GitHub, check if there is a Pull + Request that can be imported; see :ref:`section-spkg-patch-from-pr` below. + +- Sometimes it may seem as if you need to patch a (hand-written) + ``Makefile`` because it "hard-codes" some paths or compiler flags: + + .. CODE-BLOCK:: diff + + --- a/Makefile + +++ b/Makefile + @@ -77,7 +77,7 @@ + # This is a Makefile. + # Handwritten. + + -DESTDIR = /usr/local + +DESTDIR = $(SAGE_ROOT)/local + BINDIR = $(DESTDIR)/bin + INCDIR = $(DESTDIR)/include + LIBDIR = $(DESTDIR)/lib + + Don't use patching for that. Makefile variables can be overridden + from the command-line. Just use the following in ``spkg-install``: + + .. CODE-BLOCK:: bash + + $(MAKE) DESTDIR="$SAGE_ROOT/local" + +- If the upstream Makefile does not build shared libraries, + don't bother trying to patch it. + + Autoconfiscate the package instead and use the standard facilities + of Automake and Libtool. This ensures that the shared library build + is portable between Linux and macOS. + +- If you have to make changes to ``configure.ac`` or other source + files of the autotools build system (or if you are autoconfiscating + the package), then you can't use patching; make a :ref:`modified + tarball ` instead. + +- If the patch would be huge, don't use patching. Make a + :ref:`modified tarball ` instead. + +- Otherwise, :ref:`maintain a set of patches + `. + + +.. _section-spkg-patch-from-pr: + +Patching sources by importing a pull request from GitHub +-------------------------------------------------------- + +Only ``normal`` packages can be patched; see :ref:`section-package-source-types`. +If a Python package is currently a ``wheel`` package +and you need to patch it, change it to a ``normal`` package first. + +In the easiest and quite commmon case, a pull request is already available on +the upstream package's GitHub repository. + +For example, if https://github.com/discopt/cmr/pull/64 is the PR that we wish to use, +change the URL to https://github.com/discopt/cmr/pull/64.patch and save this file +in the ``patches/`` subdirectory of the package directory (create the subdirectory +if it does not exist yet). Make sure that it has the ``.patch`` +file name extension; if your browser saved it with a ``.patch.txt`` extension, +rename it. + +Modify the ``package-version.txt`` file to indicate the changed patch level; see +:ref:`section-spkg-versioning`. This ensures that the package will be rebuilt, +even though its upstream version did not change. This is important because when +you open a pull request in the Sage repository that adds the patch, also +our automatic test runs on GitHub Actions need to rebuild the packages, +and likewise when other people are testing your pull request. + +Next, test building the package with the patch, for example using ``make build``. +You should see a message like ``Applying 64.patch``. Messages such as +``Hunk #1 succeeded at 144 with fuzz 1 (offset 9 lines)`` are safe to +ignore. They appear when the PR from which you prepared the patch is based +on a version that differs from the version that the Sage package uses, or +when there are other patches that make changes to the same file. + + +Manual preparation of patches +----------------------------- + +Patches must include documentation in their header (before the first diff hunk), and +must have only one "prefix" level in the paths (that is, only one path level +above the root of the upstream sources being patched). So a typical patch file +should look like this: + +.. CODE-BLOCK:: diff + + Add autodoc_builtin_argspec config option + + Following the title line you can add a multi-line description of + what the patch does, where you got it from if you did not write it + yourself, if they are platform specific, if they should be pushed + upstream, etc... + + diff -dru Sphinx-1.2.2/sphinx/ext/autodoc.py.orig Sphinx-1.2.2/sphinx/ext/autodoc.py + --- Sphinx-1.2.2/sphinx/ext/autodoc.py.orig 2014-03-02 20:38:09.000000000 +1300 + +++ Sphinx-1.2.2/sphinx/ext/autodoc.py 2014-10-19 23:02:09.000000000 +1300 + @@ -1452,6 +1462,7 @@ + + app.add_config_value('autoclass_content', 'class', True) + app.add_config_value('autodoc_member_order', 'alphabetic', True) + + app.add_config_value('autodoc_builtin_argspec', None, True) + app.add_config_value('autodoc_default_flags', [], True) + app.add_config_value('autodoc_docstring_signature', True, True) + app.add_event('autodoc-process-docstring') + +Patches directly under the ``patches/`` directly are applied automatically +before running the ``spkg-install`` script (so long as they have the ``.patch`` +extension). If you need to apply patches conditionally (such as only on +a specifically platform), you can place those patches in a subdirectory of +``patches/`` and apply them manually using the ``sage-apply-patches`` script. +For example, considering the layout: + +.. CODE-BLOCK:: text + + SAGE_ROOT/build/pkgs/foo + |-- patches + | |-- solaris + | | |-- solaris.patch + | |-- bar.patch + | `-- baz.patch + +The patches ``bar.patch`` and ``baz.patch`` are applied to the unpacked +upstream sources in ``src/`` before running ``spkg-install``. To conditionally +apply the patch for Solaris the ``spkg-install`` should contain a section like +this: + +.. CODE-BLOCK:: bash + + if [ $UNAME == "SunOS" ]; then + sage-apply-patches -d solaris + fi + +where the ``-d`` flag applies all patches in the ``solaris/`` subdirectory of +the main ``patches/`` directory. + + +.. _section-spkg-patch-maintenance: + +How to maintain a set of patches +-------------------------------- + +We recommend the following workflow for maintaining a set of patches. + +- Fork the package and put it on a public git repository. + + If upstream has a public version control repository, import it from + there. If upstream does not have a public version control + repository, import the current sources from the upstream tarball. + Let's call the branch ``upstream``. + +- Create a branch for the changes necessary for Sage, let's call it + ``sage_package_VERSION``, where ``version`` is the upstream version + number. + +- Make the changes and commit them to the branch. + +- Generate the patches against the ``upstream`` branch: + + .. CODE-BLOCK:: bash + + rm -Rf SAGE_ROOT/build/pkgs/PACKAGE/patches + mkdir SAGE_ROOT/build/pkgs/PACKAGE/patches + git format-patch -o SAGE_ROOT/build/pkgs/PACKAGE/patches/ upstream + +- Optionally, create an ``spkg-src`` file in the Sage package's + directory that regenerates the patch directory using the above + commands. + +- When a new upstream version becomes available, merge (or import) it + into ``upstream``, then create a new branch and rebase it on top of + the updated upstream: + + .. CODE-BLOCK:: bash + + git checkout sage_package_OLDVERSION + git checkout -b sage_package_NEWVERSION + git rebase upstream + + Then regenerate the patches. + + +.. _section-spkg-src: + +Modified tarballs +----------------- + +If you really must modify the upstream tarball, then it is +recommended that you write a script, called ``spkg-src``, that makes the +changes. This not only serves as documentation but also makes it easier +to apply the same modifications to future versions. From 51fddd84824d746ded4abd0f92f677babf5704a1 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Wed, 27 Mar 2024 21:16:50 -0700 Subject: [PATCH 2/3] src/doc/en/developer/packaging.rst: Suggested changes --- src/doc/en/developer/packaging.rst | 37 +++++++++++++++++------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/src/doc/en/developer/packaging.rst b/src/doc/en/developer/packaging.rst index 14b83d6701c..5d1fce231eb 100644 --- a/src/doc/en/developer/packaging.rst +++ b/src/doc/en/developer/packaging.rst @@ -127,7 +127,7 @@ the following source types: - the file ``package-version.txt`` is optional; - - may be associated with a source tree included in the repository; + - may be associated with a source tree in the repository; - installing the package runs the installation script ``spkg-install`` or ``spkg-install.in`` (see :ref:`section-spkg-install`); @@ -368,8 +368,8 @@ at build time, which should to the appropriate system-specific If an ``spkg-src`` file is present, it indicates that the tarball is not an unmodified third-party tarball (see :ref:`section-spkg-patching`). -It documents how the tarball was generated (either by changing an upstream -tarball or generating it from an repository). As ideally +It documents how the tarball was generated (either by modifying an upstream +tarball or generating it from a repository). As ideally our tarballs are not modified, for most packages there is no ``spkg-src`` file. @@ -1214,10 +1214,13 @@ Modifying third-party code In the Sage distribution, we try to use unpatched original upstream tarballs of stable versions of third-party packages whenever possible. - Sometimes, however, modifications are necessary, either to fix a bug or to make the package build on the platforms supported by Sage. +Only ``normal`` packages can be patched; see :ref:`section-package-source-types`. +If a Python package is currently a ``wheel`` package +and you need to patch it, change it to a ``normal`` package first. + .. _section-spkg-patch-or-repackage: @@ -1278,12 +1281,8 @@ When to patch, when to repackage, when to autoconfiscate .. _section-spkg-patch-from-pr: -Patching sources by importing a pull request from GitHub --------------------------------------------------------- - -Only ``normal`` packages can be patched; see :ref:`section-package-source-types`. -If a Python package is currently a ``wheel`` package -and you need to patch it, change it to a ``normal`` package first. +Preparing a patch by importing a pull request from GitHub +--------------------------------------------------------- In the easiest and quite commmon case, a pull request is already available on the upstream package's GitHub repository. @@ -1297,10 +1296,8 @@ rename it. Modify the ``package-version.txt`` file to indicate the changed patch level; see :ref:`section-spkg-versioning`. This ensures that the package will be rebuilt, -even though its upstream version did not change. This is important because when -you open a pull request in the Sage repository that adds the patch, also -our automatic test runs on GitHub Actions need to rebuild the packages, -and likewise when other people are testing your pull request. +even though its upstream version did not change. This is important in particular +when other people are testing your added patch. Next, test building the package with the patch, for example using ``make build``. You should see a message like ``Applying 64.patch``. Messages such as @@ -1309,9 +1306,17 @@ ignore. They appear when the PR from which you prepared the patch is based on a version that differs from the version that the Sage package uses, or when there are other patches that make changes to the same file. +Be sure add the patch file to your branch using ``git add``. When you commit it, +use a commit message such as +``build/pkgs/cmr: Add https://github.com/discopt/cmr/pull/64 as a patch``. +When you open your PR from this branch, our automatic test runs on GitHub +Actions will automatically rebuild the patched package. + + +.. _section-spkg-patch-manually: -Manual preparation of patches ------------------------------ +Preparing a patch manually +-------------------------- Patches must include documentation in their header (before the first diff hunk), and must have only one "prefix" level in the paths (that is, only one path level From a69a4664c08b98501fe21af7038bc06873c1c0a0 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Wed, 27 Mar 2024 21:21:43 -0700 Subject: [PATCH 3/3] src/doc/en/developer/packaging.rst: Move 'inclusion procedure' to the end --- src/doc/en/developer/packaging.rst | 178 ++++++++++++++--------------- 1 file changed, 89 insertions(+), 89 deletions(-) diff --git a/src/doc/en/developer/packaging.rst b/src/doc/en/developer/packaging.rst index 5d1fce231eb..f9e45dd2ffb 100644 --- a/src/doc/en/developer/packaging.rst +++ b/src/doc/en/developer/packaging.rst @@ -1118,95 +1118,6 @@ If all went fine, open a PR with the code under ``SAGE_ROOT/build/pkgs``. -.. _section-inclusion-procedure: - -Inclusion procedure for new and updated packages -================================================ - -Packages that are not part of Sage will first become optional or -experimental (the latter if they will not build on all supported -systems). After they have been in optional for some time without -problems they can be proposed to be included as standard packages in -Sage. - -To propose a package for optional/experimental inclusion please open a GitHub -PR added with labels ``c: packages: experimental`` or ``c: packages: -optional``. The associated code requirements are described in the following -sections. - -After the PR was reviewed and included, optional packages stay in -that status for at least a year, after which they can be proposed to be -included as standard packages in Sage. For this a GitHub PR is opened -with the label ``c: packages: standard``. Then make -a proposal in the Google Group ``sage-devel``. - -Upgrading packages to new upstream versions or with additional patches -includes opening a PR in the respective category too, as described -above. - -License information -------------------- - -If you are patching a standard Sage spkg, then you should make sure that -the license information for that package is up-to-date, both in its -``SPKG.rst`` or ``SPKG.txt`` file and in the file ``SAGE_ROOT/COPYING.txt``. For -example, if you are producing an spkg which upgrades the vanilla source -to a new version, check whether the license changed between versions. - -If an upstream tarball of a package cannot be redistributed for license -reasons, rename it to include the string ``do-not-distribute``. This -will keep the release management scripts from uploading it to the Sage mirrors. - -Sometimes an upstream tarball contains some distributable parts using -a free software license and some non-free parts. In this case, it can -be a good solution to make a custom tarball consisting of only the free -parts; see :ref:`section-spkg-src` and the ``giac`` package as an example. - - -Prerequisites for new standard packages ---------------------------------------- - -For a package to become part of Sage's standard distribution, it -must meet the following requirements: - -- **License**. For standard packages, the license must be compatible - with the GNU General Public License, version 3. The Free Software - Foundation maintains a long list of `licenses and comments about - them `_. - -- **Build Support**. The code must build on all the fully supported - platforms (Linux, macOS); see :ref:`chapter-portability_testing`. - It must be installed either from source as a normal package, - or as a Python (platform-independent) wheel package, see - :ref:`section-package-source-types`. - -- **Quality**. The code should be "better" than any other available - code (that passes the two above criteria), and the authors need to - justify this. The comparison should be made to both Python and other - software. Criteria in passing the quality test include: - - - Speed - - - Documentation - - - Usability - - - Absence of memory leaks - - - Maintainable - - - Portability - - - Reasonable build time, size, dependencies - -- **Previously an optional package**. A new standard package must have - spent some time as an optional package. Or have a good reason why - this is not possible. - -- **Refereeing**. The code must be refereed, as discussed in - :ref:`chapter-github`. - - .. _section-spkg-patching: Modifying third-party code @@ -1429,3 +1340,92 @@ If you really must modify the upstream tarball, then it is recommended that you write a script, called ``spkg-src``, that makes the changes. This not only serves as documentation but also makes it easier to apply the same modifications to future versions. + + +.. _section-inclusion-procedure: + +Inclusion procedure for new and updated packages +================================================ + +Packages that are not part of Sage will first become optional or +experimental (the latter if they will not build on all supported +systems). After they have been in optional for some time without +problems they can be proposed to be included as standard packages in +Sage. + +To propose a package for optional/experimental inclusion please open a GitHub +PR added with labels ``c: packages: experimental`` or ``c: packages: +optional``. The associated code requirements are described in the following +sections. + +After the PR was reviewed and included, optional packages stay in +that status for at least a year, after which they can be proposed to be +included as standard packages in Sage. For this a GitHub PR is opened +with the label ``c: packages: standard``. Then make +a proposal in the Google Group ``sage-devel``. + +Upgrading packages to new upstream versions or with additional patches +includes opening a PR in the respective category too, as described +above. + +License information +------------------- + +If you are patching a standard Sage spkg, then you should make sure that +the license information for that package is up-to-date, both in its +``SPKG.rst`` or ``SPKG.txt`` file and in the file ``SAGE_ROOT/COPYING.txt``. For +example, if you are producing an spkg which upgrades the vanilla source +to a new version, check whether the license changed between versions. + +If an upstream tarball of a package cannot be redistributed for license +reasons, rename it to include the string ``do-not-distribute``. This +will keep the release management scripts from uploading it to the Sage mirrors. + +Sometimes an upstream tarball contains some distributable parts using +a free software license and some non-free parts. In this case, it can +be a good solution to make a custom tarball consisting of only the free +parts; see :ref:`section-spkg-src` and the ``giac`` package as an example. + + +Prerequisites for new standard packages +--------------------------------------- + +For a package to become part of Sage's standard distribution, it +must meet the following requirements: + +- **License**. For standard packages, the license must be compatible + with the GNU General Public License, version 3. The Free Software + Foundation maintains a long list of `licenses and comments about + them `_. + +- **Build Support**. The code must build on all the fully supported + platforms (Linux, macOS); see :ref:`chapter-portability_testing`. + It must be installed either from source as a normal package, + or as a Python (platform-independent) wheel package, see + :ref:`section-package-source-types`. + +- **Quality**. The code should be "better" than any other available + code (that passes the two above criteria), and the authors need to + justify this. The comparison should be made to both Python and other + software. Criteria in passing the quality test include: + + - Speed + + - Documentation + + - Usability + + - Absence of memory leaks + + - Maintainable + + - Portability + + - Reasonable build time, size, dependencies + +- **Previously an optional package**. A new standard package must have + spent some time as an optional package. Or have a good reason why + this is not possible. + +- **Refereeing**. The code must be refereed, as discussed in + :ref:`chapter-github`.