Change the command to initialize DDP subprocess with Hydra

[zihaozou]

Description & Motivation

Hi,

the current behavior to initialize DDP subprocesses with Hydra is here, which overrides the hydra.run.dir with os.getcwd().

This is assuming the os.getcwd() will return the output directory that Hydra assigned. However, since Hydra 1.2, Hydra will keep the cwd as the project directory by default. This command will result in logging the subprocess in the project directory.

Pitch

Logging the DDP subprocess correctly in the output directory created by Hydra

Alternatives

use:

os_cwd = hydra.core.hydra_config.HydraConfig.get()['runtime']['output_dir']
os_cwd = f'"{os_cwd}"'

instead

Additional context

No response

cc @Borda @justusschock @awaelchli

[awaelchli]

@zihaozou Thanks for the request.

I remember we had changed the behavior once around this (#11617) but had to revert it (#15737).
It looks like this is about the same thing, but I'm not sure.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions - the Lightning Team!