sklearn preprocessor output in 'application/x-recordio-protobuf' fails with either import error or version conflict

[SarathChandran]

System Information

• Framework (e.g. Sklearn) / Algorithm (e.g. Linear-Learner):
• Framework Version:
• Python Version: 3.5
• CPU or GPU: CPU
• Python SDK Version:
• Are you using a custom image: No

Describe the problem

When trying to preprocess input data with sklearn framework and save the output as application/x-recordio-protobuf, I am facing import issues with 'sagemaker' module.

ImportError: No module named 'sagemaker'

If i install sagemaker via os.system('pip install sagemaker') in the preprocess script. I still get a version conflict on boto.

**Collecting sagemaker **
Downloading https://files.pythonhosted.org/packages/8f/38/fa369edea6817971e354594b25ccb75d9f8186445417a07f851416d75b47/sagemaker-1.18.16.tar.gz (188kB)

** Collecting boto3>=1.9.64 (from sagemaker) **
Downloading https://files.pythonhosted.org/packages/3f/1b/4adde85d1cbbe8a8f2fef47276daf496328fc8d7c40d7a8d6a67b0eba45b/boto3-1.9.135-py2.py3-none-any.whl (128kB)

** sagemaker_containers._errors.ClientError: (boto3 1.9.51 (/usr/local/lib/python3.5/dist-packages), Requirement.parse('boto3>=1.9.64'), {'sagemaker'}) **

Since i could not get access to the container, i am puzzled about the problem.

Steps to reproduce:
In Example: Inference Pipeline with Scikit-learn and Linear Learner, include 'accept == application/x-recordio-protobuf'

elif accept == 'application/x-recordio-protobuf':
# os.system("pip install sagemaker --upgrade")
# os.system("pip install boto3 --upgrade")
import sagemaker.amazon.common as smac fails to read this import
buf = BytesIO()
print(prediction)
print(prediction.shape)
smac.write_spmatrix_to_sparse_tensor(buf, prediction)
buf.seek(0)
return worker.Response(buf, accept, mimetype=accept)

[jesterhazy]

Hi, @SarathChandran,

Thanks for using SageMaker!

It looks like you are trying to install our client-side SDK into your sklearn container during your training job. This isn't a supported usage, and isn't usually a useful thing to do anyway.

The reason your pip install commands aren't working is because the older boto3 version is already imported into the current python process. Changing the package installed on the file system only affects the next python program launched.

Right now there are no helper functions for writing recordio available in the sci-kit learn container. You would need to implement your own -- perhaps by copying the implementation you found in sagemaker.amazon.common into your own inference script.

[SarathChandran]

@jesterhazy , thanks for the response. But this is kind of a workaround. Isn't the sci-kit learn support to contain sagemaker dependencies by default? Also, there is no way to specify version inside the container.

Could you please take this as a feature request?

[jesterhazy]

The sagemaker python package is meant to be a client-side library that lets you start training jobs, endpoints, etc. It isn't intended to be used from inside running sagemaker jobs.

I think what is missing here is support for recordio and sparse matrices within the sklearn container. I'll close this issue, and open new one in the correct repo.

[SarathChandran]

@jesterhazy could you share the open issue url here

[vanetoj]

Did this ever resolve? I have written 4 different models and it keeps failing to rin on the sagemaker cuz they cant put clear directions with the data conversion. . not happy !

[laurenyu]

the open issue can be found at aws/sagemaker-scikit-learn-container#10