Race condition in matplotlib mkdir()

[jdemeyer]

I regularly see this error when testing on the buildbot:

sage -t -long  -force_lib devel/sage/doc/de/tutorial/tour_functions.rst
**********************************************************************
File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/devel/sage-main/doc/de/tutorial/tour_functions.rst", line 24:
    sage: plot(f, 0, 2)
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_0[5]>", line 1, in <module>
        plot(f, Integer(0), Integer(2))###line 24:
    sage: plot(f, 0, 2)
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/sage/misc/displayhook.py", line 174, in displayhook
        print_obj(sys.stdout, obj)
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/sage/misc/displayhook.py", line 142, in print_obj
        print >>out_stream, `obj`
      File "sage_object.pyx", line 154, in sage.structure.sage_object.SageObject.__repr__ (sage/structure/sage_object.c:1463)
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/sage/plot/plot.py", line 1081, in _repr_
        self.show()
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/sage/misc/decorators.py", line 426, in wrapper
        return func(*args, **kwds)
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/sage/plot/plot.py", line 1723, in show
        self.save(DOCTEST_MODE_FILE, **kwds)
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/sage/misc/decorators.py", line 426, in wrapper
        return func(*args, **kwds)
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/sage/plot/plot.py", line 2453, in save
        figure = self.matplotlib(**options)
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/sage/plot/plot.py", line 1930, in matplotlib
        from matplotlib.figure import Figure, figaspect
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/matplotlib/figure.py", line 18, in <module>
        from axes import Axes, SubplotBase, subplot_class_factory
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/matplotlib/axes.py", line 18, in <module>
        import matplotlib.contour as mcontour
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/matplotlib/contour.py", line 21, in <module>
        import matplotlib.texmanager as texmanager
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/matplotlib/texmanager.py", line 72, in <module>
        class TexManager:
      File "/Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/build/sage-4.5.3-4.7.1/local/lib/python/site-packages/matplotlib/texmanager.py", line 92, in TexManager
        os.mkdir(texcache)
    OSError: [Errno 17] File exists: '/tmp/dot_sage.WboXjWq0ow/matplotlib-1.0.1/tex.cache'

The problem looks to be the following race condition in the matplotlib code:

    if not os.path.exists(texcache):
        os.mkdir(texcache)

---

New spkg available at

http://sage.math.washington.edu/home/palmieri/SPKG/matplotlib-1.0.1.p0.spkg

CC: @jhpalmieri @jasongrout

Component: packages: standard

Keywords: MPL Errno 17 libpng

Author: John Palmieri

Reviewer: Leif Leonhardy

Merged: sage-4.7.2.alpha2

Issue created by migration from https://trac.sagemath.org/ticket/11686

[nexttime]

comment:1

Seen this before (i.e., similar), haven't we?

[jhpalmieri]

comment:2

Looks like the same issue (with the same fix) as in #10159. I reported this to the matplotlib mailing list in October 2010, but no response.

In fact, what happened to the patches from #10159 which dealt with this issue? Leif has noted on #10588 that those patches seem to have been lost in the update to version 1.0.1...

[nexttime]

comment:3

Replying to @nexttime:

Seen this before (i.e., similar), haven't we?

This is a regression introduced by #10588, previously fixed in John's spkg from #10159, because the spkg of the former had not been based on the latter (and the bug hasn't [yet] been fixed upstream, at least not in 1.0.1).

Also, the current spkg is full of crap in src/build/. (The whole directory can be deleted.)

[nexttime]

comment:4

Replying to @jhpalmieri:

In fact, what happened to the patches from #10159 which dealt with this issue? Leif has noted on #10588 that those patches seem to have been lost in the update to version 1.0.1...

It's different to what I first thought; see my previous comment... ;-)

With a bit of luck your patches from #10159 still apply seamlessly to the current upstream version; haven't tested that yet.

[nexttime]

comment:5

Just checked: John's 1.0.0.p0 spkg has been in Sage 4.6.1 and 4.6.2.alpha0, then the one from #10588 got merged into Sage 4.6.2.alpha1.

[nexttime]

comment:6

Replying to @nexttime:

With a bit of luck your patches from #10159 still apply seamlessly to the current upstream version; haven't tested that yet.

Hmmm, a little work has to be done:

~/Sage/spkgs/matplotlib-1.0.1.p1/patches$ patch __init__.py < __init__.py.patch 
patching file __init__.py
Hunk #1 succeeded at 103 with fuzz 2.
~/Sage/spkgs/matplotlib-1.0.1.p1/patches$ patch finance.py < finance.py.patch 
patching file finance.py
Hunk #1 FAILED at 4.
Hunk #2 succeeded at 173 (offset 4 lines).
1 out of 2 hunks FAILED -- saving rejects to file finance.py.rej
~/Sage/spkgs/matplotlib-1.0.1.p1/patches$ patch texmanager.py < texmanager.py.patch 
patching file texmanager.py
$

John, are you going to do this?

[jhpalmieri]

Author: John Palmieri

[jhpalmieri]

Description changed:

--- 
+++ 
@@ -52,3 +52,10 @@
     if not os.path.exists(texcache):
         os.mkdir(texcache)
 ```
+
+---
+
+New spkg available at
+
+[http://sage.math.washington.edu/home/palmieri/SPKG/matplotlib-1.0.1.p0.spkg](http://sage.math.washington.edu/home/palmieri/SPKG/matplotlib-1.0.1.p0.spkg)
+

[jhpalmieri]

comment:7

I've created a new spkg, link in the ticket description. I'm posting the diff here for review purposes.

[jhpalmieri]

comment:8

(I also deleted src/build and a few random .pyc files.)