Meta-ticket: support macOS 11 Big Sur

[slel]

This is to support macOS 11 Big Sur.

• switch the default mp library to gmp #30752 gmp instead of mpir
• GH Actions: Update macos test environments #30929 GH Actions: Update macos test environments
• Allow using arb, flint and ntl from Homebrew #30745 homebrew: Errors when homebrew's ntl, flint, or ecl are installed in /usr/local
• Upgrade: wheel 0.36.2 #31050 upgrade wheel
• MACOSX_DEPLOYMENT_TARGET problems - https://groups.google.com/g/sage-devel/c/Sj3YBVJoHQM/m/Zt-Ih_hJAwAJ
• MR50: Upgrade cffi to 1.14.4 #31128 MR50: Upgrade cffi to 1.14.4
• Update setuptools and setuptools_scm #31134 upgrade setuptools and setuptools_scm
• Upgrade Python to 3.9.1, pip to 20.3.3 #30589 Upgrade Python to 3.9.1, pip to 20.3.3
• numpy: Fix for macOS 11 (Big Sur) #31166 numpy: Fix for macOS 11 (Big Sur)
• Fix scipy for MacOS 11 (Big Sur) #31183 scipy: fix for macOS 11 (Big Sur)
• macos-11.0: scipy build fails #31326 homebrew-macos-11.0-xcode-minimal: scipy build fails

Related:

• Meta-ticket: Support Xcode 12 #30494: Meta-ticket: Support Xcode 12
• Meta-ticket: Port to Apple silicon #30592: Meta-ticket: Port to Apple silicon

References:

• https://bugs.python.org/issue41116
• https://groups.google.com/g/sage-devel/c/5lBTr4gZuic/m/RkisHYLWCgAJ

Depends on #30494

CC: @dimpase @jhpalmieri @mkoeppe @slel @vbraun @williamstein

Component: porting

Keywords: macOS

Issue created by migration from https://trac.sagemath.org/ticket/30651

[mkoeppe]

Description changed:

--- 
+++ 
@@ -5,3 +5,8 @@
 Related:
 
 - #30494: Meta-ticket: Support Xcode 12
+
+References:
+- https://bugs.python.org/issue41116
+
+

[wdjoyner]

comment:3

I'm trying to compile sage-9.2.rc0 on big sur with xcode 12.0.1.

I ran sage -f openblas. Here's the tail of that:

(see the openblas attachment).

After trying lots of things such as

$ ./configure \
  --enable-universalsdk=$(xcodebuild -version -sdk macosx Path) \
  --with-universal-archs=intel-64
$ source /Users/wdj/sagefiles/sage-9.2.rc0/.homebrew-build-env
$ ./config.status --recheck && ./config.status
$ ./sage -f gfortran

the compilation of sage -f gfortran stopped with an error compiling R.

However, Sage did start and crash with this error:

wdj@jeeves sage-9.2.rc0 % ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.2.rc0, Release Date: 2020-10-05                 │
│ Using Python 3.8.5. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
---------------------------------------------------------------------------
ModuleNotFoundErrorPython 3.8.5:
/Users/wdj/sagefiles/sage-9.2.rc0/local/bin/python3
Thu Oct  8 13:27:16 2020
A problem occurred executing Python code.  Here is the sequence of function
calls leading up to the error, with the most recent (innermost) call last.

(for the rest, see the error-in-start attachment).

It looks like Sage's Python doesn't work, right? Wrong:

wdj@jeeves sage-9.2.rc0 % ./sage -python3
Python 3.8.5 (default, Oct  7 2020, 13:59:53)
[Clang 12.0.0 (clang-1200.0.32.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 2+3
5

In case it's any use, here is what homebrew claims it has installed:

wdj@jeeves sage-9.2.rc0 % brew list --versions
autoconf 2.69
automake 1.16.1_1
bdw-gc 8.0.4
boost 1.71.0
cmake 3.15.5
fontconfig 2.13.1
freetype 2.10.1
gcc 9.2.0_1
gd 2.2.5
gettext 0.20.1 0.21
gmp 6.2.0 6.1.2_2
gnutls 3.6.15
gpatch 2.7.6
guile 3.0.4
icu4c 64.2
isl 0.22.1 0.21
jpeg 9c
libde265 1.0.3
libdvdcss 1.4.2
libevent 2.1.12
libffi 3.3 3.2.1
libidn2 2.3.0 2.2.0_1
libmpc 1.1.0 1.2.0
libomp 9.0.0
libpng 1.6.37
libtasn1 4.14 4.16.0
libtiff 4.0.10_1
libtool 2.4.6_1 2.4.6_2
libunistring 0.9.10
libyaml 0.2.5
mpfi 1.5.3
mpfr 4.1.0 4.0.2
nettle 3.4.1 3.6
ntl 11.4.1
openssl 1.0.2t
openssl@1.1 1.1.1g 1.1.1d
p11-kit 0.23.21 0.23.18.1
pari 2.11.4
pkg-config 0.29.2_3 0.29.2
readline 8.0.4
ruby 2.7.1_2
sqlite 3.33.0
unbound 1.11.0
webp 1.0.3
xz 5.2.4

[jhpalmieri]

comment:4

I've actually gotten this to build for me without patching Sage in any way, just building 9.2.rc0. Homebrew setup:

% brew list
autoconf    gd           haskell-stack  libpng        ncurses      r
automake    gdbm         icu4c          libtasn1      nettle       re2c
bdw-gc      gettext      ilmbase        libtiff       ninja        readline
bison       ghc          intltool       libtool       openblas     sphinx-doc
boost       ghostscript  isl            libunistring  openexr      sqlite
bzip2       git          jansson        libxml2       openssl@1.1  suite-sparse
cmake       glib         jpeg           meson         p11-kit      tbb
curl        glpk         libatomic_ops  metis         pandoc       texinfo
doxygen     gmp          libevent       mpc           pcre         unbound
fontconfig  gnutls       libffi         mpfi          pcre2        webp
freetype    gpatch       libidn2        mpfr          pkg-config   xz
gcc         gsl          libmpc         mpir          ppl          yasm
gcc@9       guile        libmpdclient   nauty         python@3.8   zeromq

Then I used the command

$ ./configure \
  --with-system-readline=no \
  --with-system-zlib=no \
  --with-system-gfortran=no \
  --with-system-openblas=no \
  --with-system-bzip2=no

after which make worked but make ptestlong
showed a bunch of failures, all like this:

Expected nothing
Got:
    ld: warning: dylib (/usr/local/lib/libgmp.dylib) was built for newer macOS version (10.15) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libmpfr.dylib) was built for newer macOS version (10.15) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgmpxx.dylib) was built for newer macOS version (10.15) than being linked (10.9)

Maybe because MACOSX_DEPLOYMENT_TARGET is being set? I could try building Sage's mpfr and gmp. But in any case, I got Sage to build, unfortunately by not using a bunch of the system versions of things.

[jhpalmieri]

comment:5

Replying to @jhpalmieri:

Maybe because MACOSX_DEPLOYMENT_TARGET is being set? I could try building Sage's mpfr and gmp.

I could try that, but it won't work because our mpir is broken when using Xcode 12, with the usual error, and the latest release of mpir (git clone git://github.com/wbhart/mpir.git mpir) doesn't help.

[dimpase]

comment:6

mpir is basically not developed any more (for the time being at least), so we should not use it.

What is the value of MACOSX_DEPLOYMENT_TARGET used in this build? I would not be surprised if there is an incompatibility barrier between macOS 10 and macOS 11.

[dimpase]

comment:7

Replying to @wdjoyner:

I'm trying to compile sage-9.2.rc0 on big sur with xcode 12.0.1.

I ran sage -f openblas. Here's the tail of that:

[...]

David, could you please replace this 10-screen comment by an attachement?

[wdjoyner]

Attachment: openblas-tail.txt

openblas error

[wdjoyner]

Attachment: error-in-start.txt

error in start

[jhpalmieri]

comment:8

Replying to @dimpase:

mpir is basically not developed any more (for the time being at least), so we should not use it.

It's a standard package, so I'm confused about what the alternatives are.

Anyway, I hacked the spkg-install.in script for mpir, as in #30729, and I got it to build. That eliminated those doctest failures. I'm left with two:

sage -t --long --random-seed=0 src/sage/misc/gperftools.py
**********************************************************************
File "src/sage/misc/gperftools.py", line 118, in sage.misc.gperftools.Profiler._libc
Failed example:
    Profiler()._libc()
Exception raised:
    Traceback (most recent call last):
      File "/Users/palmieri/Sage/sage-9.2.rc0/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 720, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/palmieri/Sage/sage-9.2.rc0/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 1145, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.gperftools.Profiler._libc[1]>", line 1, in <module>
        Profiler()._libc()
      File "/Users/palmieri/Sage/sage-9.2.rc0/local/lib/python3.8/site-packages/sage/misc/gperftools.py", line 129, in _libc
        raise ImportError('failed to open libc')
    ImportError: failed to open libc

and

sage -t --long --random-seed=0 src/sage/tests/parigp.py
    Timed out

What is the value of MACOSX_DEPLOYMENT_TARGET used in this build? I would not be surprised if there is an incompatibility barrier between macOS 10 and macOS 11.

I haven't changed Sage, so it's whatever gets set in sage-env: 10.9, I think.