merge cypari and cypari2

[videlec]

The original cypari written by N. Dunfield and M. Culler will be merged with this project to ease maintenance. Though we are facing several incompatibilities

1 - Windows

cypari can be installed on Windows (however not directly from source, they provide wheels on PyPI). This is at the cost of a fork of cysignals and many special casing.

• cysignals is not working on windows. See issue 63.
• issues with long (cypari defines a new pari_ulongword type)

Note that these issues refer to plain Windows. Under cygwin, there are no such issues.

2 - Static vs dynamic linking / source vs distribution

cypari currently ships gmp, pari and (a fork of) cysignals. cypari2 contains only sources for a Python module. The original cypari has the advantage of being installable on many platforms using wheels (among other it solves the problem of having cysignals built after libpari).

We propose in the merged version to have three possible ways of installation

1. pip install cypari: installs a wheel with static linking and embedded cysignals (i.e. what is currently available with the cypari wheels)

2. pip install --no-binary cypari: installs from an sdist with dynamic linking from gmp/pari on the system and dependency from cysignals (i.e. the cypari2 version)

3. (optional) pip install --no-binary --install-option=static-link cypari: installs from an sdist but also builds libpari.a and libgmp.a and statically links against them (intermediate solution)

4 - Cython macro

Cython macros (either defined using DEF or via compile_time_env option of cythonize) modifies the C code generated by Cython from the pyx files. Having many of them imposes to provide a different versions of the generated C files for each possible values of the constants (in order to allow to build from source without Cython).

The Python2/Python3 compatibility can be achieved without any of them (as done in cypari2, see #13). However it seems unavoidable for the Windows port.

[jdemeyer]

Building gmp and pari would better not be part of the setup.py script.

In my view, this is mostly an problem of user interface. It's trivial to support different kinds of builds. But the hard question is: how does the interface work? What should happen when somebody does pip install cypari? When installing from source using setup.py, it would be possible to require an argument like setup.py --build-pari or whatever. But with wheels, I don't think this is possible.

[videlec]

Building gmp and pari would better not be part of the setup.py script.

In my view, this is mostly an problem of user interface. It's trivial to support different kinds of builds. But the hard question is: how does the interface work? What should happen when somebody does pip install cypari? When installing from source using setup.py, it would be possible to require an argument like setup.py --build-pari or whatever. But with wheels, I don't think this is possible.

This is why I proposed in the exchange of e-mails to have two packages cypari-mini (only source?) and cypari-full (source and wheels). I don't think there is a need for more options. Don't you like this way?

[NathanDunfield]

@videlec With regards to cypari-mini versus cypari-full is the idea that they would both install a package named cypari but they would be separate packages on PyPI? Or would they install differently named packages? Either way, we probably need to think about what should happen when pip tries to install another Python package that depends on cypari-*.

[videlec]

With regards to cypari-mini versus cypari-full is the idea that they would both install a package named cypari but they would be separate packages on PyPI? Or would they install differently named packages? Either way, we probably need to think about what should happen when pip tries to install another Python package that depends on cypari-*.

@NathanDunfield my idea is not completely clear yet :-) I definitely want both versions to install a cypari package. The best I think will be to have cypari-full as a "virtual package" that do the install gmp, pari, cysignals, cypari-mini in that order. If this is the way to go, we would perhaps better use the name cypari for cypari-mini because it would match the Python module names. However, this change would break backward compatibility of pip install cypari.

Concerning dependencies on cypari, is it possible to have fine dependencies in pip? I imagine something like: "if cypari-mini is there then fine, otherwise install cypari-full".

[culler]

@videlec @NathanDunfield

A simple implementation of this idea might be to just have a setup.py option which specifies whether to use system libraries or to build the libraries from scratch. The default would be to use system libraries. That way:
pip install --no-binary cypari
would be the equivalent of installing cypari-mini (provided the build succeeds). On the other hand
pip install cypari
would download the wheel, which would necessarily include its own copies of the libraries, so that would be the equivalent of installing cypari-full. To build the wheels one would supply the option to setup.py which indicates to build the libraries.
Someone who wanted to build their own standalone module could just clone the repo.

[NathanDunfield]

@videlec wrote:

Concerning dependencies on cypari, is it possible to have fine dependencies in pip? I imagine something like: "if cypari-mini is there then fine, otherwise install cypari-full".

You can achieve this sort of effect by putting some logic in the dependent package's setup.py script and modifying the list of packages passed to setup(install_requires=...). We do this with SnapPy, e.g. not including cypari as a requirement when installing into Sage. But so far I haven't seen any built in support for this in pip, e.g. requirements of the form foo>2.0|bar>1.5 which is satisfied if either is present and otherwise it tries to install the first one.

@culler wrote:

On the other hand pip install cypari would download the wheel, which would necessarily include its own copies of the libraries, so that would be the equivalent of installing cypari-full.

One possible issue: support for Linux wheels in pip is pretty new, and so on some distros pip install cypari would actually try to build the source tarball (unless one also posts eggs).

[NathanDunfield]

@videlec @culler
Another approach would be to have three packages: cypari, cypari-full and cypari-mini. The first contains only a few lines of code that try to import cypari-mini and cypari-full in turn; when it succeeds it sets cypari.pari to cypari_found.pari etc and otherwise raises an ImportError. If you do pip install cypari, then we make this succeed only if one of cypari-full and cypari-mini is present, otherwise it throws polite error message indicating that the user should first install their choice of cypari-full and cypari-mini. We could put in certain shortcuts, e.g. on Windows pip install cypari would automatically require cypari-full. Possibly the same thing should be true on macOS for a Frameworks build of Python (such as the ones downloaded from Python.org) as opposed the "plain unix style" Pythons that come with SageMath and (I think) brew.

For example, one concrete possibility would be to have the setup.py script of cypari do the following:

1. Try to import cypari-mini and cypari-full. If one is available and has the same version number as cypari then everything is good and it just installs its own few lines of code.

2. If no suitable version of cypari-* is available and the OS is either Windows or macOS with a Frameworks build of Python, then add cypari-full to install_requirements and pip will handle the rest.

3. On Linux or with a "plain unix" macOS python, then it could do some tests to see if there appears to be suitable versions of gmp and pari available. If so, add cypari-mini (which forces cysignals) to the install_requirements and otherwise add cypari-full.

[videlec]

@NathanDunfield

Another approach would be to have three packages: cypari, cypari-full and cypari-mini.
[ ... snip ...]

With your proposal, if I want the minimal version should I do

$ pip install cypari-mini
$ pip install cypari

3. On Linux or with a "plain unix" macOS python, then it could do some tests to see if there appears to be suitable versions of gmp and pari available. If so, add cypari-mini (which forces cysignals) to the install_requirements and otherwise add cypari-full.

It is hard to provide reliable tests to see whether one can build cypari-mini. Namely, the user might not have gcc. I would be more convinced by a solution that by default always install cypari-full unless explicitely specified (as Marc proposed with --no-binary).

[NathanDunfield]

@videlec wrote:

With your proposal, if I want the minimal version should I do

$ pip install cypari-mini
$ pip install cypari

Yes, that's right.

It is hard to provide reliable tests to see whether one can build cypari-mini. Namely, the user might not have gcc. I would be more convinced by a solution that by default always install cypari-full unless explicitly specified (as Marc proposed with --no-binary).

I agree that figuring out whether cypari-mini can be installed is tricky and should be avoided if possible. I would be happy with a solution that installs cypari-full unless explicitly specified (via --no-binary or some other flag), but wasn't sure if that would be acceptable to you guys.

[NathanDunfield]

Also, I was thinking that perhaps we should use slightly more descriptive names, perhaps cypari-selfcontained and cypari-system-pari or cypari-static and cypari-libpari.

[videlec]

Then, we might want to stick to a unique cypari package when in source mode (i.e. --no-binary pip option) will use whatever pari is available on the system (that is the current cypari2 version) and for which all the wheels are self-contained (ie the current cypari wheels)? When used in source mode it is assumed that libpari is available and that can be tested for providing nice error message.

I am not sure there is a need for intermediate installations. This is the job of distributions.

I see one backward compatibility with this proposal for people who used to install the original cypari in source mode.

[NathanDunfield]

So pip has an --install-option flag which can be used to pass options down to setup.py. It seems like this would allow us to cover the backward compatibility case you mention via something like:

pip install --no-binary --install-option=build-own-pari  cypari

Putting this together, a concrete plan would be:

1. Have a single cypari package on PyPI with the the usual 16 or so cypari-full binary wheels and a source tarball containing the Cythonized C source files but not the gmp and pari sources.

2. Have the source tarball default to cypari-mini unless a suitable flag is set, in which case it fetches the gmp and pari sources and builds its own copy of pari from scratch.

What do people think of such a plan? It seems good to me.

[culler]

I see one backward compatibility with this proposal for people who used to install the original cypari in source mode.

I think Nathan, Matthias and I will be able to handle it. :^)
Oops, I forgot Bill Allombert! He will be OK too.

[culler]

That sounds great to me! - Marc

…

On Fri, Jul 21, 2017 at 10:46 AM, NathanDunfield ***@***.***> wrote: So pip has an --install-option flag which can be used to pass options down to setup.py. It seems like this would allow us to cover the backward compatibility case you mention via something like: pip install --no-binary --install-option=build-own-pari cypari Putting this together, a concrete plan would be: 1. Have a single cypari package on PyPI with the the usual 16 or so cypari-full binary wheels and a source tarball containing the Cythonized C source files but not the gmp and pari sources. 2. Have the source tarball default to cypari-mini unless a suitable flag is set, in which case it fetches the gmp and pari sources and builds its own copy of pari from scratch. What do people think of such a plan? It seems good to me. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABPhP0YkawD8XN10vYkqjgQxCj61gZDAks5sQMfkgaJpZM4Od2xy> .