python2/3: split spkg-install to spkg-build; fix various build issues

[embray]

This ticket starts by splitting the spkg-install for python2 and python3 into separate spkg-build and spkg-install scripts as recommended by #21726 (we said we would apply this change to individual packages on a case-by-case basis, and I figured as long as I was making changes to these scripts it would be good to do).

This also fixes a general bug with the build, where building a new Python can link the python executable and extension modules with an old libpython left in $SAGE_LOCAL, leading to possible errors.

Rather than deleting the old libpython first, I think it's cleaner to pass the correct $LDFLAGS to prevent this from happening.

Finally, this moves the tests that certain extension modules built to after building, but before installing, and runs the tests out of the source directory. This is a change extracted from #22509.

Component: packages: standard

Keywords: python2 python3

Author: Erik Bray

Branch/Commit: f326c1a

Reviewer: Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/23781

[jdemeyer]

comment:2

Given that these scripts are mostly the same for Python 2 and Python 3, would it be possible to have just one script (say, by symlinking) and then use $PKG_NAME inside the script where needed?

[jdemeyer]

comment:3

Looking at the difference between the Python 2 and Python 3 scripts, I wonder why this only occurs for Python 2:

  # Use EXTRA_CFLAGS for user-defined CFLAGS since Python puts its own
  # default flags like -O3 after CFLAGS but before EXTRA_CFLAGS.
  # We also disable warnings about unused variables/functions which are
  # common in Cython-generated code.
  export EXTRA_CFLAGS="`testcflags.sh -Wno-unused` $CFLAGS"
  unset CFLAGS

I think this is just an omission and it should be there for Python 3 also.

[jdemeyer]

comment:4

Apart from that, everything looks fine from looking at the diff.

[embray]

comment:5

Replying to @jdemeyer:

Given that these scripts are mostly the same for Python 2 and Python 3, would it be possible to have just one script (say, by symlinking) and then use $PKG_NAME inside the script where needed?

I've wondered that myself. It's tempting--maybe this ticket would be a good place to do it as long as I'm making changes...

[embray]

comment:6

Replying to @jdemeyer:

Looking at the difference between the Python 2 and Python 3 scripts, I wonder why this only occurs for Python 2:

  # Use EXTRA_CFLAGS for user-defined CFLAGS since Python puts its own
  # default flags like -O3 after CFLAGS but before EXTRA_CFLAGS.
  # We also disable warnings about unused variables/functions which are
  # common in Cython-generated code.
  export EXTRA_CFLAGS="`testcflags.sh -Wno-unused` $CFLAGS"
  unset CFLAGS

I think this is just an omission and it should be there for Python 3 also.

Apparently you added that in #19899, but didn't add it to python3, presumably just due to lack of python3 testing at the time. Probably it applies there too.

[jdemeyer]

comment:7

Replying to @embray:

Replying to @jdemeyer:

Given that these scripts are mostly the same for Python 2 and Python 3, would it be possible to have just one script (say, by symlinking) and then use $PKG_NAME inside the script where needed?

I've wondered that myself. It's tempting--maybe this ticket would be a good place to do it as long as I'm making changes...

Great! Please do.

[embray]

comment:8

For some reason I thought I already did this. I guess I'll finish this up now since this is a dependency of #22509, which I'd also really like to move forward on.