Infrastructure for conversions of Sage number types for external Cython/Python packages (fpylll, pypolymake, ...)

[mkoeppe]

There is a current trend to develop Cython libraries as separate packages, rather than as part of Sage.

Many of these libraries take numbers of various kinds as input or produce them as output.

It would be nice if these Cython libraries could do conversions from/to Sage number types. Otherwise, on the Sage side we would have to provide a separate wrapper around the Cython wrapper.

Currently, this leads to

• awkward dependencies (such as fpylll depending on sagelib - see this sage-devel discussion and Make Sage run-time dependency of fpylll #21728) and to
• duplication of effort (for example, PyPolymake and its GMP wrappers)

We need to develop shared infrastructure (a new lightweight library, which does NOT depend on sagelib) for this.

CC: @malb @jdemeyer @videlec @infinity0 @mezzarobba @sagetrac-tmonteil @dimpase @slel @defeo @w-bruns

Component: interfaces

Reviewer: Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/21725

[jdemeyer]

Replying to @mkoeppe:

We need to develop shared infrastructure (a new lightweight library, which does NOT depend on sagelib) for this.

That's not going to happen.

[mezzarobba]

comment:3

If a Python call + a copy per number conversion it not too high a price to pay, then perhaps sage objects backed by a “well-known” C library could provide an interface that would make them read/write an mpz_t (resp. mpq_t, mpfr_t, mpfi_t...) from/to a specified location? This would at least make it possible to exchange multiple-precision numbers with sage without build-depending on it.

[mkoeppe]

comment:4

Replying to @jdemeyer:

Replying to @mkoeppe:

We need to develop shared infrastructure (a new lightweight library, which does NOT depend on sagelib) for this.

That's not going to happen.

How do you know?

[jdemeyer]

comment:6

I mean, how could it ever be possible to convert something to Sage types (I assume you mean with an efficient C interface) without depending on Sage?

[jdemeyer]

comment:7

Replying to @mezzarobba:

If a Python call + a copy per number conversion it not too high a price to pay, then perhaps sage objects backed by a “well-known” C library could provide an interface that would make them read/write an mpz_t (resp. mpq_t, mpfr_t, mpfi_t...) from/to a specified location?

Sage types already have such an interface using Cython. So you want to add something like add using a Python interface? I presume using a PyCapsule?

To be honest, I don't really understand the use case for this ticket. Why would you want to convert something to Sage without depending on Sage?

[jdemeyer]

Replying to @mkoeppe:

• duplication of effort (for example, PyPolymake and its GMP wrappers)

Can you elaborate on this?

[mkoeppe]

comment:9

For example, it would not seem so hard to provide a C library with a minimal interface such as

int mpz_from_python_integer(mpz_t target, PyObject *object)

that fills target with a GMP integer converted from object
and returns an error code if it can't.

By default (in plain Python), it would only deal with Python integers.

But libraries that provide new integer types such as sagelib can hook into this, for example by defining a method or any other of Python's dynamic mechanisms.

For the opposite conversion, for converting to the Python/Sage world, one would pass an extra argument to the wrapped function that indicates the type of output (Python integer? Sage integer? ...)

[mkoeppe]

comment:10

Replying to @jdemeyer:

Replying to @mkoeppe:

• duplication of effort (for example, PyPolymake and its GMP wrappers)

Can you elaborate on this?

Various libraries that I am currently interested in (polymake, normaliz) need Cython wrappers, which should be usable with or without Sage. The interfaces are using integers (and integer vectors, matrices, and more complicated structures) - and one should be able to request Python integers or Sage integers from the Cython wrapper.
However, it should definitely not build-depend on sagelib (because we will need the opposite dependency -- sagelib will have to build-depend on pyPolymake, pyNormaliz etc.)

[mkoeppe]

comment:11

Replying to @mkoeppe:

For the opposite conversion, for converting to the Python/Sage world, one would pass an extra argument to the wrapped function that indicates the type of output (Python integer? Sage integer? ...)

For example C library prototype:

PyObject *mpz_to_python_integer(mpz_t source, PyObject *output_type)

[videlec]

comment:12

Replying to @mkoeppe:

Replying to @mkoeppe:

For the opposite conversion, for converting to the Python/Sage world, one would pass an extra argument to the wrapped function that indicates the type of output (Python integer? Sage integer? ...)

For example C library prototype:

PyObject *mpz_to_python_integer(mpz_t source, PyObject *output_type)

This is exactly the kind of thing I am fighting with in pypolymake. Actually pypolymake could easily provide a function to set the interface

int set_pypolymake_mpz_wrapper((PyObject *) (mpz_t, PyObject *))

I am not sure what is the best approach here but +1 for having a standardized and Sage agnostic way of doing it! I am available to test as many approaches as suggested within pypolymake.

Note that we might want to do that for many basic types (integer, rational, floating point real and complexes, Zmodn, matrices, polynomials, ...). And this explosion might become a nightmare.

[jdemeyer]

comment:13

Replying to @mkoeppe:

For example, it would not seem so hard to provide a C library

I hope that you mean a Python module and not a C library. Libraries are a pain in Python.

One thing we really need to consider is efficiency. We already have plenty of slow ways to do what you want (e.g. for integers we can go via Python ints, for other types we could use strings). So I would like a solution which is implementable 100% in Cython (in particular, there should be no Python methods/attributes involved).

[jdemeyer]

comment:15

There is this thing: https://pypi.python.org/pypi/gmpy2 which provides an interface to GMP and MPFR. It implements wrappers around the GMP/MPFR types, so it could probably be used to pass around mpz_t pointers.

I don't think it was created with this use case in mind, but I still think that this could be a good starting point. We obviously would need to add a Cython interface.