symlink not accepted as logger folder

[jowagner]

🐛 Bug

Moving lightning_logs to another filesystem and symlinking it so I don't need to touch my code, I notice that the logger rejects symlinked logger folders:

loggers/csv_logs.py if not os.path.isdir(root_dir):

Users will expect symlinks to folders to work the same way as folders.

To Reproduce

mkdir /tmp/lightning_logs
ln -s /tmp/lightning_logs/

and try training as usual.

Expected behavior

The symlinked folder is accepted and, in the above example, the logs start filling /tmp instead of $HOME.

Workarounds

(Breaking existing paths)

Solution 1: Create a folder inside the symlinked folder and set save_dir to this subfolder (not tested)
Solution 2: Use the absolute path to the folder, e.g. trainer = pl.Trainer(logger = pl.loggers.TensorBoardLogger(os.path.abspath('lightning_logs'))) (I was expecting the next run to go to version_3 after I had versions 0, 1 and 2 but it went to default/version_0 so there still seems to be a side effect)

Environment

Private, local Jupyter notebook with manually setup virtualenv.

• PyTorch Version (e.g., 1.0): 1.7.1+cu101
• OS (e.g., Linux): Linux
• How you installed PyTorch (conda, pip, source): pip
• Build command you used (if compiling from source):
• Python version: 3.6.12
• CUDA/cuDNN version: 10.1
• GPU models and configuration: GeForce GTX Titan-X
• Any other relevant information: None

Additional context

None

[ethanwharris]

Hi @jowagner, thanks for the issue! I had a go at CSV logging to a symlink and it is working fine at least for me. The os.path.isdir docs seem to suggest that it follows symlinks, so shouldn't be an issue. Can you maybe reproduce the bug in colab using the BoringModel? You can use os.symlink to create a symlink between some folders on the colab storage 😃

It may also be worth checking if the bug is still there if you update pytorch-lightning to master using the instructions here: https://github.com/PyTorchLightning/pytorch-lightning#install-bleeding-edge---future-12

[jowagner]

Right. Commenting out my workaround logger= ... abspath ... lines, l cannot reproduce the error. This is in a new jupyter server instance as I had killed my server in the meantime. Maybe there are some strange interactions when moving the folder to another filesystem while the jupyter kernel is running. (The notebook was idle.) I'll need more time to try to reproduce the error. It was the "Missing logger folder" message. I now see there is a second occurrence of this message with a slightly different condition:

$ find -type f -name "*.py" | xargs -d'\n' fgrep -A 2 -B 1 "Missing logger folder" | tail -n 4
./loggers/tensorboard.py-        if not self._fs.isdir(root_dir):
./loggers/tensorboard.py:            log.warning('Missing logger folder: %s', root_dir)
./loggers/tensorboard.py-            return 0
./loggers/tensorboard.py-

I find self._fs = get_filesystem(...) and this leads to a call of fsspec.filesystem(), which may be https://pypi.org/project/fsspec/ and may have different semantics of isdir(). Though that cannot really be it given that the symlink now works in a new jupyter server instance. Maybe fsspec does some caching of metadata and doesn't behave well when a dir changes to a symlink. Will need more time to try to reproduce the error to find out more.

[tchaton]

Dear @jowagner,

Please, keep us updated as symlink should definitely work !

Best,
T.C

[jowagner]

I cleaned out the folder the symlink points to and restarted the notebook server. Training runs through OK but the "Missing logger folder: /home/retracted/notebooks/lightning_logs" message is still there. Repeat runs of the notebook write to the same "version_0" folder. This means it is not just a problem of a warning message but there is a change of behaviour.

The following suggests that the isdir() semantics for fsspec are indeed different:

$ ls -l
[...]
lrwxrwxrwx 1 retracted users         32 Mar 26 17:09 lightning_logs -> /scratch/retracted/lightning_logs/
[...]
$ python
Python 3.6.12 (default, Dec 02 2020, 09:44:23) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fsspec
>>> fs = fsspec.filesystem('file')
>>> fs.isdir('lightning_logs')
False

[ethanwharris]

@jowagner, thanks for providing some more information. That definitely lools like a bug, I'll do a bit of digging and try to work on a fix 😃