EIA deploy error no libcuda.so.1

[austinmw]

I'm trying to deploy a TF Script Mode trained model that uses TF Serving to a CPU+EIA, but the endpoint is failing to be created. The logs say:

tensorflow_model_server: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory

So does TF Serving require CUDA Toolkit? and If so is this toolkit not installed on SM official cpu/eia docker images?

Edit: Actually this now seems like a duplicate of a previous bug I posted. Working backwards, the model created from estimator.deploy lists the image pulled as the GPU version: 520713654638.dkr.ecr.us-west-2.amazonaws.com/sagemaker-tensorflow-serving:1.12.0-gpu even though I specified a cpu instance type.

[ChoiByungWook]

Hello @austinmw,

Sorry for the late response.

Can you post your Python SDK code that you use to run your SageMaker job?

If I understand you correctly, it seems the EI CPU image is failing out with the error message `libcuda.so.1:' above or the GPU image is actually being pulled down instead of the EI image.

I apologize for the inconvenience.

[austinmw]

@ChoiByungWook If I do:

train_instance_type =  'ml.p3.2xlarge'

estimator = TensorFlow(entry_point='launcher.sh',
                       source_dir='src',
                       train_instance_type=train_instance_type,
                       train_instance_count=1,
                       hyperparameters=hyperparameters,
                       role=role,
                       framework_version='1.12',
                       py_version='py3', 
                       script_mode=True,
                       output_path=output_path,
                       dependencies=[],
                       metric_definitions=metric_definitions)

estimator.fit(inputs)

predictor =  estimator.deploy(initial_instance_count=1,
                               instance_type='ml.c5.xlarge', 
                               accelerator_type='ml.eia1.large')

Deploying fails with that error. Checking the image it pulled under console shows the gpu version. I have to do this instead:

    from sagemaker.tensorflow.serving import Model

    model = Model(model_data=estimator.model_data,
                  role=sagemaker.get_execution_role(),
                  image=None,
                  framework_version='1.12')

    predictor = model.deploy(endpoint_name=endpoint_name,
                                          initial_instance_count=1,
                                          instance_type='ml.c5.xlarge',
                                          accelerator_type='ml.eia1.large')

[ChoiByungWook]

@austinmw,

Are you running in a Jupyter notebook?

What version of the Python SDK are you using?

I'm having some trouble reproducing this. I debugged this with pdb and I am getting proper CPU images back.

[ChoiByungWook]

I tried reproducing this with this notebook and am having trouble: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/tensorflow_iris_dnn_classifier_using_estimators/tensorflow_iris_dnn_classifier_using_estimators_elastic_inference.ipynb

I replaced the instance type to ml.p3.2xlarge for the training instance and passed in a ml.c5.xlarge for the deploy.

I checked the training-job and it pulled the gpu image and also checked the model and endpoint. The correct image was pulled for both. Attaching my images.

Training job details

SageMaker Model attached to my endpoint

I am really interested in root causing what could be causing the issue you ran into.

[ChoiByungWook]

@austinmw,

I believe I found the root cause now.

The script_mode flag being True ends up with the base Model object being created instead of the TensorFlowModel object, which leads to the proper construction of the ECR image.

https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/estimator.py#L407

This leads to this code being called that generates the GPU ECR image you saw pointing to the tensorflow-serving-container.

https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/serving.py#L125

I don't think this is the proper user experience we should be giving at all and I apologize for that. Let me relay this back to the team and try to get a resolution.

Thank you for reporting this.

[austinmw]

@ChoiByungWook sorry for the late reply. Awesome that you were able to track down the issue, thanks!

[ChoiByungWook]

For reference:

#609
https://stackoverflow.com/questions/54485769/sagemaker-deploying-to-eia-from-tf-script-mode-python3

[austinmw]

Added an edit clarifying that I also needed to use sagemaker.tensorflow.serving.Model instead of sagemaker.tensorflow.TensorFlowModel

[ChoiByungWook]

For reference: aws/sagemaker-python-sdk@4732545#r32336726

[ChoiByungWook]

Wanted to clarify for those that are stumbling on this thread that there are always non-EIA supported versions of the same framework in SageMaker. This can be used in the Python SDK by not specifying an accelerator_type when calling deploy.

For more information on Elastic Inference (EI) please see: https://docs.aws.amazon.com/sagemaker/latest/dg/ei.html