Meta-ticket: replace pexpect interface calls with library where possible

[rwst]

This meta-ticket tracks efforts to replace pexpect interface calls with library calls, in order to minimize the usage of expect for time-critical code. Packages used by Sage where this seems possible include

• mwrank
• GAP (convert all of Sage library code to libgap #26902)
• Singular (no ticket but see related libSingular slower than Singular-via-pexpect #11957)
• ECL/Maxima (move pexpect-maxima code to maxima_lib #17753)
• PARI/GP (a few scripts Use gp2c for GP scripts #16687)

The following packages are used via pexpect in the Sage library (noninteractively) and these calls should be replaced somehow in
the long term (this is a wishlist):

• Axiom (through fricas)
• giac (first step: make giacpy_sage standard: Make giacpy_sage a standard package #28918, second step: integration use libgiac instead of giac for integration #31873)
• kash
• Macaulay2
• Octave
• qepcad
• Scilab (two unused functions in matrix/matrix1.pyx)

Related:

• Alert pexpect interfaces to library interfaces #22680: Alert pexpect interfaces to library interfaces
• Pexpect-like interfaces based on jupyter_client #22579: Pexpect-like interfaces based on jupyter_client

CC: @kiwifb @frederichan-IMJPRG @nexttime @slel

Component: interfaces

Keywords: expect, pexpect, library

Issue created by migration from https://trac.sagemath.org/ticket/16688

[kiwifb]

comment:1

The notebook also uses pexpect. I am not sure there is much of an alternative there but at least we should try to move it to a more modern pexpect. Trying to do so results in breakages that I have no idea how to solve currently.

[williamstein]

comment:3

Further motivation for this:

sage: scilab(str(range(20)))  # works
sage: scilab(str(range(30)))  # BOOM!

(Note -- scilab(range(20)) doesn't work but should!)

And also

sage: r(range(30))  # works
sage: r(range(10000)) # boom

Worksheet with the above: https://cloud.sagemath.com/projects/4a5f0542-5873-4eed-a85c-a18c706e8bcd/files/support/2015-02-04-062417-scilab-limits.sagews

These sorts of problems were reported by many people over the years, most recently by JC Boulet of INRA.

[jdemeyer]

comment:4

I would say it's a motivation to improve the pexpect interfaces (see for example #17704, #17718, #17686, #17719).

I claim that such interfaces can be made to work, it's just a mess in Sage.

[nbruin]

comment:5

Replying to @kiwifb:

The notebook also uses pexpect. I am not sure there is much of an alternative there but at least we should try to move it to a more modern pexpect. Trying to do so results in breakages that I have no idea how to solve currently.

There is, especially when you're programming both sides. You can have two processes communicate through a pipe or stream without one of the parties pretending to be a terminal.

That said, the fact that the notebook does use pexpect makes it easier to use it to interface with R, octave, magma, etc, so we do get some benefit from it.

[williamstein]

comment:6

Replying to @nbruin:

Replying to @kiwifb:

The notebook also uses pexpect. I am not sure there is much of an alternative there but at least we should try to move it to a more modern pexpect. Trying to do so results in breakages that I have no idea how to solve currently.

There is, especially when you're programming both sides. You can have two processes communicate through a pipe or stream without one of the parties pretending to be a terminal.

I completely agree. IPython and also https://github.com/sagemath/cloud/blob/master/sage_server.py are both examples of better ways of communication between two processes without using pexpect. Basically, you setup a traditional client/server and a messaging protocol. Python and many other languages makes that sort of thing pretty easy. But... it would be harder to implement something like that in a special purpose system like Pari or Magma. It might be impossible with Magma without source code access (not sure what the status of networking is now).

That said, the fact that the notebook does use pexpect makes it easier to use it to interface with R, octave, magma, etc, so we do get some benefit from it.

As the author of exactly the part of the notebook that uses pexpect, etc., I don't agree with this statement so much -- instead, the two are somewhat orthogonal. The connection (1) "notebook_server <--> python processes" is done using pexpect in sagenb. The interfaces to R/octave/magma, etc. are connections (2) "python processes <--> R/octave/magma/etc.". You can certainly swap out (1) for a connection built using TCP (rather than pexpect) with no impact on (2); in fact, that's precisely what I did with SageMathCloud.

That said, maybe you just meant that work on the abstract base class Expect for the interfaces benefits both sagenb and the R/octave/magma, etc., interfaces...

[jdemeyer]

comment:7

I do agree that when alternatives to pexpect are available, we should use the alternatives. But for the programs which don't offer alternatives, pexpect should be able to work.

[rwst]

Description changed:

--- 
+++ 
@@ -3,7 +3,7 @@
 * mwrank
 * GAP
 * Singular
-* ECL/Maxima
+* ECL/Maxima (#17753)
 * PARI/GP (a few scripts #16687)
 
 The following packages are used via pexpect in the Sage library (noninteractively) and these calls should be replaced somehow in

[slel]

Description changed:

--- 
+++ 
@@ -1,7 +1,7 @@
 This meta-ticket tracks efforts to replace pexpect interface calls with library calls, in order to minimize the usage of expect for time-critical code. Packages used by Sage where this seems possible include
 
 * mwrank
-* GAP
+* GAP (#26902)
 * Singular
 * ECL/Maxima (#17753)
 * PARI/GP (a few scripts #16687)

[slel]

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@
 the long term (this is a wishlist):
 
 * Axiom (through fricas)
-* giac
+* giac (first step: make `giacpy_sage` standard: #28918)
 * kash
 * Macaulay2
 * Octave

[slel]

Description changed:

--- 
+++ 
@@ -2,7 +2,7 @@
 
 * mwrank
 * GAP (#26902)
-* Singular
+* Singular (no ticket but see related #11957)
 * ECL/Maxima (#17753)
 * PARI/GP (a few scripts #16687)
 

[slel]

Description changed:

--- 
+++ 
@@ -16,3 +16,9 @@
 * Octave
 * qepcad
 * Scilab (two unused functions in `matrix/matrix1.pyx`)
+
+Related:
+
+- #22680: Alert pexpect interfaces to library interfaces
+- #22579: Pexpect-like interfaces based on jupyter_client
+