Pathlib compat fails under Python 3.5

[khaeru]

pytest's compatibility code chooses to use pathlib2 except for Python 3.6. This means that pathlib2.Path objects are returned for fixtures/fixture methods like tmp_path_factory.mktemp(). When the user tests code using Python 3.5's built-in pathlib, the built-in package chokes on the objects from pathlib2.

Put the following in test_foo.py and execute using pytest under Python 3.5:

from pathlib import Path  # available in Python 3.5


def foo(p):
    p2 = p / 'foo.txt'
    print(p2.exists())


def bar(p):
    p2 = Path(p)
    print(p2.exists())


def test_foo(tmp_path_factory):
    foo(tmp_path_factory.mktemp('baz1'))


def test_bar(tmp_path_factory):
    bar(tmp_path_factory.mktemp('baz2'))

…this occurs:

$ pytest test_foo.py
============================================================================== test session starts ===============================================================================
platform linux -- Python 3.5.2, pytest-4.4.0, py-1.5.2, pluggy-0.9.0
Matplotlib: 2.1.2
Freetype: 2.6.1
rootdir: /home/gidden/work/iiasa/message/ixmp, inifile: setup.cfg
plugins: mpl-0.9, cov-2.6.0
collected 2 items
                                                                                                                                                                
test_foo.py .F                                                                                                                                                             [100%]

==================================================================================== FAILURES ====================================================================================
____________________________________________________________________________________ test_bar ____________________________________________________________________________________
tmp_path_factory = TempPathFactory(_given_basetemp=None, _trace=<pluggy._tracing.TagTracerSub object at 0x7f0779c3f1d0>, _basetemp=PosixPath('/tmp/pytest-of-gidden/pytest-22'))
    def test_bar(tmp_path_factory):
    
>       bar(tmp_path_factory.mktemp('baz2'))
test_foo.py:24: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_foo.py:13: in bar
    p2 = Path(p)
/usr/lib/python3.5/pathlib.py:969: in __new__
    self = cls._from_parts(args, init=False)
/usr/lib/python3.5/pathlib.py:651: in _from_parts
    drv, root, parts = self._parse_args(args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cls = <class 'pathlib.PosixPath'>, args = (PosixPath('/tmp/pytest-of-gidden/pytest-22/baz20'),)
    @classmethod
    def _parse_args(cls, args):
        # This is useful when you don't want to create an instance, just
        # canonicalize some constructor arguments.
        parts = []
        for a in args:
            if isinstance(a, PurePath):
                parts += a._parts
            elif isinstance(a, str):
                # Force-cast str subclasses to str (issue #21127)
                parts.append(str(a))
            else:
                raise TypeError(
                    "argument should be a path or str object, not %r"
>                   % type(a))
E               TypeError: argument should be a path or str object, not <class 'pathlib2.PosixPath'>
/usr/lib/python3.5/pathlib.py:643: TypeError
======================================================================= 1 failed, 1 passed in 0.13 seconds =======================================================================

The same command succeeds under Python 3.6.

Discovered by @gidden

[nicoddemus]

Hi @khaeru, thanks for the report.

We decided to use pathlib2 on py<3.6 because the standard pathlib was missing a few methods IIRC. @RonnyPfannschmidt knows the details.

A workaround is to use Path(str(p)) instead of Path(p), but I believe you already know that.

[khaeru]

Hi @nicoddemus — yes, that's what we're doing for now.

However, it was somewhat cryptic, so still took us some sleuthing to figure out was going on. Only when the pytest-returned pathlib2 objects hit pathlib code (e.g. "bar" in the snippet above) does this happen; in other cases ("foo") it does not.

For less experienced pytest users, it might help to:

• explicitly state in the documentation which of pathlib/pathlib2 is used, and/or
• catch/re-raise with an explanatory message/hint.

[RonnyPfannschmidt]

Its a unfortunate fact that pathlib pre python 3.6 is a broken mess

[nicoddemus]

I will see if we can drop pathlib2 on Python 3; I think we can live with the missing bits in Python 3.4 and 3.5.