Python 3.6: Fix locale/encoding issues in docbuild, then re-enable Python 3.6

[mkoeppe]

Follow-up from #15791, #29033, #30053, the latter of which disabled use of system Python 3.6.

With Python 3.6, there is a failure building the documentation on ubuntu-bionic-standard (using /usr/bin/python3.6, https://github.com/mkoeppe/sage/runs/1106251169):

  [dochtml]   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2661: ordinal not in range(128)
  [dochtml] Full log file: logs/dochtml.log
Makefile:1876: recipe for target 'doc-html' failed

This is coming from #30002 "force C locale in docbuild and remove some obsolete stuff", which added this:

build/bin/sage-site:    export LANG=C # to ensure it is possible to scrape out non-EN locale warnings

In the present ticket, we rework the fix in #30002 so that it works with Python 3.6 too (in which setting the "C" locale sabotages UTF-8 operation.)

Supporting Python 3.6 allows Sage to use the system Python on some older Linux distributions that are still in widespread use in scientific computing, including centos-8 and fedora-{26,27,28} (although Python 3.7.x packages are also available for these). See #29033 for more details.

For the unrelated failures in the Sphinx package installation that #29033 tried to work around, see #30008.
(#30008 is a dependency of this ticket merely for the convenience of testing.)

---

References:

• https://www.python.org/dev/peps/pep-0538/ "Coercing the legacy C locale to a UTF-8 based locale"
• https://www.python.org/dev/peps/pep-0540/ "Add a new UTF-8 mode"
• https://bugzilla.redhat.com/attachment.cgi?id=1233034&action=diff

Depends on #30008

CC: @antonio-rojas @dimpase @slel @orlitzky @kiwifb @embray @fchapoton

Component: build

Author: Matthias Koeppe

Branch/Commit: 945c8c5

Reviewer: Sébastien Labbé

Issue created by migration from https://trac.sagemath.org/ticket/30576

[seblabbe]

comment:1

Posting my branch containing few changes that need to be done in order to build doc with ubuntu 18.04 bionic and python 3.6.

The changes are not sufficient. Other similar issues arise when sphinx is running...

---

New commits:

e09d256

setting few utf-8 encodings

[seblabbe]

Branch: public/30576

[seblabbe]

Commit: e09d256

[seblabbe]

comment:2

Two comments here:

• fixing the issues in the posted branch took a while because there are many try - except clauses in the doc-building which eat up all of the useful tracebacks.
• I think the remaining issues are just WARNINGS that breaks the docbuild because we decided at some point in the past to consider WARNINGS as errors (which is fine). I base this observation on some hacky modifications I did in the src/sage_setup/docbuild/sphinxbuild.py. To confirm this, do you know if there is a way to build sage documentation by considering warnings just as warnings?

[dimpase]

comment:3

Warnings one keeps getting are errors in processing function arguments, i.e. in introspecting Python code. Ignoring them would give you a broken doc.

error while formatting arguments for sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.complete_primary_decomposition: 'ascii' codec can't decode byte 0xc2 in position 4774: ordinal not in range(128)

I think the problems come from a new(ish) sphinx which is run as a backend for our custom docbuilder, and anyway the move from Python 3.6 to 3.7 was somewhat big as far as utf-8 support is concerned.

I really think that fully supporting 3.6 is hard, and we have better things to do. Python has solved this problem by moving to 3.7, let's not reinvent the wheel here.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,6 +1,6 @@
-Follow-up from #29033, #30053.
+Follow-up from #29033, #30053, the latter of which disabled use of system Python 3.6.
 
-And a failure building the documentation on `ubuntu-bionic-standard` (using `/usr/bin/python3.6`, https://github.com/mkoeppe/sage/runs/1106251169):
+With Python 3.6, there is a failure building the documentation on `ubuntu-bionic-standard` (using `/usr/bin/python3.6`, https://github.com/mkoeppe/sage/runs/1106251169):
 
 ```
   [dochtml]   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2661: ordinal not in range(128)
@@ -8,4 +8,10 @@
 Makefile:1876: recipe for target 'doc-html' failed
 ```
 
+Also, there are failures in package installs that #29033 tried to work around. More complete fixes are discussed but not implemented in #30053.
 
+
+Supporting Python 3.6 allows Sage to use the system Python on some older Linux distributions that are still in widespread use in scientific computing, including centos-8 and fedora-{26,27,28} (although Python 3.7.x packages are also available for these). See #29033 for more details.
+
+
+

[mkoeppe]

Work Issues: rebase on #30053, revert commit be47518

[sagetrac-git]

Changed commit from e09d256 to 9943f25

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e5f6663	only use locale C.UTF-8 if available, else C
ff6f4a6	assume Python 3.7 or better
64c3a8a	only test python3, to be 3.7 or 3.8
be47518	build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'
af0a956	Merge branch 't/30053/build/careful_with_C_UTF8' into t/30576/public/30576
8adc360	Revert "build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'"
161765b	Revert "only test python3, to be 3.7 or 3.8"
c75a21b	setting few utf-8 encodings
9943f25	Revert "assume Python 3.7 or better"

[mkoeppe]

Dependencies: #30053

[mkoeppe]

Description changed:

--- 
+++ 
@@ -13,5 +13,10 @@
 
 Supporting Python 3.6 allows Sage to use the system Python on some older Linux distributions that are still in widespread use in scientific computing, including centos-8 and fedora-{26,27,28} (although Python 3.7.x packages are also available for these). See #29033 for more details.
 
+---
+
+References:
+- https://www.python.org/dev/peps/pep-0538/ "Coercing the legacy C locale to a UTF-8 based locale"
 
 
+

[mkoeppe]

Changed work issues from rebase on #30053, revert commit be47518 to none

[mkoeppe]

Description changed:

--- 
+++ 
@@ -17,6 +17,7 @@
 
 References:
 - https://www.python.org/dev/peps/pep-0538/ "Coercing the legacy C locale to a UTF-8 based locale"
+- https://www.python.org/dev/peps/pep-0540/ "Add a new UTF-8 mode"
 
 
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -18,6 +18,6 @@
 References:
 - https://www.python.org/dev/peps/pep-0538/ "Coercing the legacy C locale to a UTF-8 based locale"
 - https://www.python.org/dev/peps/pep-0540/ "Add a new UTF-8 mode"
+- https://bugzilla.redhat.com/attachment.cgi?id=1233034&action=diff
 
 
-

[mkoeppe]

Description changed:

--- 
+++ 
@@ -10,6 +10,21 @@
 
 Also, there are failures in package installs that #29033 tried to work around. More complete fixes are discussed but not implemented in #30053.
 
+There are a few places that explicitly set locale variables (`LANG` and `LC_*`):
+
+```
+build/bin/sage-site:    export LANG=C # to ensure it is possible to scrape out non-EN locale warnings
+build/bin/sage-spkg: export LC_ALL=C;
+build/pkgs/gap/spkg-check.in:export LC_CTYPE=en_US.UTF-8
+build/pkgs/pari/spkg-install.in:LANG=C
+build/tox.ini:    LC_ALL = C
+docker/Dockerfile:ENV LC_ALL C.UTF-8
+src/sage/docs/conf.py:locale.setlocale(locale.LC_NUMERIC, 'C')
+src/sage/interfaces/gap.py:            sage: gap = Gap(env={'LC_CTYPE': 'en_US.UTF-8'})
+src/sage/interfaces/jmoldata.py:            env['LC_ALL'] = 'C'
+```
+
+
 
 Supporting Python 3.6 allows Sage to use the system Python on some older Linux distributions that are still in widespread use in scientific computing, including centos-8 and fedora-{26,27,28} (although Python 3.7.x packages are also available for these). See #29033 for more details.