doctests: Filter out linker warning messages

[mkoeppe]

(from #30589)

On OS X Big Sur + various homebrew packages, if I use

% ./configure --with-system-python3=no --enable-download-from-upstream-url 

then Sage builds (if I also incorporate various other changes from #30651), but I get doctest failures of the form

File "src/sage/misc/sagedoc.py", line 646, in sage.misc.sagedoc.format
Failed example:
    cython('\n'.join(cython_code))
Expected nothing
Got:
    ld: warning: dylib (/usr/local/lib/libmpfr.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgmp.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgmpxx.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libgsl.dylib) was built for newer macOS version (11.0) than being linked (10.9)
    ld: warning: dylib (/usr/local/lib/libntl.dylib) was built for newer macOS version (11.0) than being linked (10.9)

We should filter out these warnings in doctesting.

CC: @jhpalmieri @dimpase @kiwifb

Component: doctest framework

Author: John Palmieri

Branch/Commit: 55cf2ee

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/31204

[jhpalmieri]

comment:1

I've only seen this on OS X. The following has worked for me, although it hasn't been thoroughly tested:

diff --git a/src/sage/doctest/parsing.py b/src/sage/doctest/parsing.py
index 4e1a6b32e4..77f726a27b 100644
--- a/src/sage/doctest/parsing.py
+++ b/src/sage/doctest/parsing.py
@@ -40,6 +40,8 @@ optional_regex = re.compile(r'(arb216|arb218|py2|py3|long time|not implemented|n
 # which has not been patched, we need to ignore that message.
 # See :trac:`29317`.
 glpk_simplex_warning_regex = re.compile(r'(Long-step dual simplex will be used)')
+# :trac:`31204` -- suppress warning about ld and OS version for dylib files.
+ld_warning_regex = re.compile(r'ld: warning: dylib.*was built for newer macOS version .* than being linked.*')
 find_sage_prompt = re.compile(r"^(\s*)sage: ", re.M)
 find_sage_continuation = re.compile(r"^(\s*)\.\.\.\.:", re.M)
 find_python_continuation = re.compile(r"^(\s*)\.\.\.([^\.])", re.M)
@@ -1087,6 +1089,7 @@ class SageOutputChecker(doctest.OutputChecker):
         """
         got = self.human_readable_escape_sequences(got)
         got = glpk_simplex_warning_regex.sub('', got)
+        got = ld_warning_regex.sub('', got)
         if isinstance(want, MarkedOutput):
             if want.random:
                 return True

I can produce a branch easily enough.

[mkoeppe]

comment:2

Looks fine to me. I briefly looked to find instances of this on Linux in GH Actions logs, but did not find anything, it's likely that I have confused this with something else.

[jhpalmieri]

comment:3

This doesn't quite catch all of the warnings:

**********************************************************************
File "src/sage/tests/cmdline.py", line 301, in sage.tests.cmdline.test_executable
Failed example:
    err
Expected:
    'Compiling ...spyx...'
Got:
    'Compiling /Users/palmieri/.sage/temp/John-iMac-2017.local/34214/dir__zddcuog/sage_test_file.spyx...\nld: warning: dylib (/usr/local/lib/libgmp.dylib) was built for newer macOS version (11.0) than being linked (10.9)\nld: warning: dylib (/usr/local/lib/libmpfr.dylib) was built for newer macOS version (11.0) than being linked (10.9)\nld: warning: dylib (/usr/local/lib/libgmpxx.dylib) was built for newer macOS version (11.0) than being linked (10.9)\nld: warning: dylib (/usr/local/lib/libgsl.dylib) was built for newer macOS version (11.0) than being linked (10.9)\nld: warning: dylib (/usr/local/lib/libntl.dylib) was built for newer macOS version (11.0) than being linked (10.9)\n'
**********************************************************************
File "src/sage/tests/cmdline.py", line 308, in sage.tests.cmdline.test_executable
Failed example:
    err
Expected:
    'Compiling ...spyx...'
Got:
    'Compiling sage_test_file.spyx...\nld: warning: dylib (/usr/local/lib/libgmp.dylib) was built for newer macOS version (11.0) than being linked (10.9)\nld: warning: dylib (/usr/local/lib/libmpfr.dylib) was built for newer macOS version (11.0) than being linked (10.9)\nld: warning: dylib (/usr/local/lib/libgmpxx.dylib) was built for newer macOS version (11.0) than being linked (10.9)\nld: warning: dylib (/usr/local/lib/libgsl.dylib) was built for newer macOS version (11.0) than being linked (10.9)\nld: warning: dylib (/usr/local/lib/libntl.dylib) was built for newer macOS version (11.0) than being linked (10.9)\n'
**********************************************************************
1 item had failures:
   2 of 231 in sage.tests.cmdline.test_executable

I'm not sure why the "..." doesn't capture all of this. Maybe it matches the verbatim "..." in the doctest output, or maybe it doesn't capture multiline strings.

[mkoeppe]

comment:4

Could you push a branch please?

[jhpalmieri]

Branch: u/jhpalmieri/31204-ld-warnings

[jhpalmieri]

comment:6

Replying to @mkoeppe:

Could you push a branch please?

Sure.

---

New commits:

bb6edf6

trac 31204: suppress ld warning messages about wrong OS X version

[jhpalmieri]

Commit: bb6edf6

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

801c1a7

trac 31204: suppress ld warning messages about wrong OS X version

[sagetrac-git]

Changed commit from bb6edf6 to 801c1a7

[jhpalmieri]

comment:8

I modified the filter a bit, starting it with "dylib..." instead of "ld: warning: dylib...", since I've seen a few cases without "ld: warning".

[zlscherr]

comment:9

I was doing some testing today and could, for example, eliminate these warnings by doing something like:

MACOSX_DEPLOYMENT_TARGET=11.0 ./sage -t --random-seed=0 src/sage/docs/instancedoc.pyx

I think the underlying problem lies in the python that sage is using. You can see this, for example, by running sage and trying

import sysconfig
sysconfig.get_config_var("MACOSX_DEPLOYMENT_TARGET")

the issue is that the homebrew libraries were built on a particular operating system with its own deployment target, which is newer than the python target of sage. This problem also manifests itself in #31227 where /usr/bin/python3 has the target set to 10.14.6 for some reason.

I wonder if you can try to fix this by making sage set MACOSX_DEPLOYMENT_TARGET environment variable when it's run. You can probably get away with just setting it to whatever version of the Mac operating system you are on.

[jhpalmieri]

comment:10

I like the idea but I'm not getting it to work:

% MACOSX_DEPLOYMENT_TARGET=11.2 ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.3.beta6, Release Date: 2021-01-17               │
│ Using Python 3.9.1. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: import sysconfig                                                                    
sage: sysconfig.get_config_var("MACOSX_DEPLOYMENT_TARGET")                                
'10.9'

Maybe part of the solution is #18272: we should not be artificially setting MACOSX_DEPLOYMENT_TARGET. That may only help when we are using Sage's own Python, but I think this problem also arises with /usr/bin/python3 or the homebrew Python 3.

[zlscherr]

comment:11

What happen's if you try

MACOSX_DEPLOYMENT_TARGET=11.0 ./sage -t src/sage/misc/sagedoc.py

That won't change sysconfig's deployment target, but it might stop the linking warnings.

(I just ran make distclean for some reason and won't be able to test until tomorrow)

[zlscherr]

comment:12

ok, that definitely didn't work. For some reason that seemed to work with /usr/bin/python3 but not with the sage built python which is built for target 10.9