MPI Smart Default for Processes per Host

[ryanh-ai]

For the referenced line/section below, would it be reasonable to have a smart default here to consume all available GPU’s per host if executed inside SM job and the parameter isn’t supplied?

Would prevent user from having to translate p3.16xlarge to 8 GPU for instance, and modifying these parameters if changing instances.

sagemaker-containers/src/sagemaker_containers/_runner.py

Line 48 in 71bd1ce

processes_per_host = _mpi_param_value(mpi_args, env, _params.MPI_PROCESSES_PER_HOST, 1)

[ChoiByungWook]

@canada4663,

Apologies, as I am not an expert with MPI.

However, this seems like a reasonable proposal. How should this value be handled if a GPU isn't being used? Would it be safe to use the number of CPUs?

I'm wondering if there are any repercussions changing from the current simple default to a smarter one for current users.

Let me reach out to the team and see what they think.

Thanks!

[armandmcqueen]

+1 for this idea. It's easy to implement since sagemaker-containers exposes the SM_NUM_GPUS environmental variable.

It seems like 1 process per host is a decent default when doing CPU training.

[ChoiByungWook]

Let me write a PR. I'll have the value of this variable follow this order:

1. customer input
2. env.num_gpus
3. 1

Whatever the customer defines as the value should override everything. If we detect GPUs we default to the number of them. Otherwise, we can assume the instance is CPU and we will default to 1.

[ChoiByungWook]

PR: #204

[ChoiByungWook]

While testing my image, it seems that the Python SDK defaults to 1 if one doesn't provide a value in the distributions in the constructor.

https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/estimator.py#L548

With this even if we do smart defaults it will go to 1.

[ChoiByungWook]

Python SDK PR to get around the default on the client side: aws/sagemaker-python-sdk#928

[ryanh-ai]

Great thanks!!!