Pytest crashes when recursive symlinks present

When I have a test directory which contains a recursive symlink (in my case this is one of the assets we are checking that our software works in such case), pytest crashes with:

E   OSError: [Errno 40] Too many levels of symbolic links: '/tmp/tests/recursive'

The /tmp/tests directory looks like the following:

zsolt@localhost:/tmp$ ls -l /tmp/tests/
total 4
lrwxrwxrwx 1 zsolt zsolt  9 Oct 27 15:02 recursive -> recursive
-rw-rw-r-- 1 zsolt zsolt 36 Oct 27 15:01 test_dummy.py
zsolt@localhost:/tmp$ cat /tmp/tests/test_dummy.py 

def test_dummy():
    assert True

It used to work with pytest 6.0.2 and now it seems to be broken on 6.1.0 and above.

I'm running pytest with pytest /tmp/tests command from its own, clean venv having the given pytest version (6.0.2, 6.1.0 and 6.1.1) and its dependencies only.
I'm using Linux (Ubuntu bionic), but I think this happens on every system supporting recursive symlinks.

[The-Compiler]

Thanks for the reproducer! Bisected to 3633b69 (pathlib: make visit() independent of py.path.local, use os.scandir from #7541) - cc @bluetech

Stacktrace:

src/_pytest/runner.py:310: in from_call
    result = func()  # type: Optional[TResult]
src/_pytest/runner.py:340: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
src/_pytest/main.py:685: in collect
    if not direntry.is_file():
E   OSError: [Errno 40] Too many levels of symbolic links: '/home/florian/proj/pytest/tests/recursive'

[csernazs]

I've added a PoC for the fix of the issue, this "works for me".
I'm not sure if it is the proper way to handle this error, maybe is_file() should be fixed by returning False for recursive symlinks (as those entries are not files for sure), however that may cause other errors.

[bluetech]

@csernazs, can you submit your patch as a PR?

The two alternatives I think are

• To catch the error and ignore it (your patch)
• To avoid following symlinks (i.e. use is_file(follow_symlinks=False) in some places).

I think your approach is probably better, but would first like to compare to whatever was the previous behavior (before 3633b69) as we might want to just emulate that.

[csernazs]

PR created. Feel free to adjust the changelog or the code.