Error hosting Sagemaker MXNet Endpoint

[nahidalam]

Please fill out the form below.

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans):
  MXNet
• Framework Version:
• Python Version:
• CPU or GPU:
• Python SDK Version:
• Are you using a custom image:

Describe the problem

I am using my own Keras pretrained model (.json, .h5) - converted to tensorflow protobuf and then gziped to model.tar.gz following this tutorial (https://aws.amazon.com/blogs/machine-learning/deploy-trained-keras-or-tensorflow-models-using-amazon-sagemaker/)

Now after putting the model.tar.gz in S3, I do below code to deploy the model

from sagemaker.mxnet import MXNetModel
mxnet_model = MXNetModel(model_data = 's3://' + sagemaker_session.default_bucket() + '/modelmxnet/model.tar.gz', role=role, entry_point='trasform_script.py')

predictor = mxnet_model.deploy(instance_type='ml.c4.xlarge', initial_instance_count=1)

When I run this cell, I get below error:
ValueError: Error hosting endpoint sagemaker-mxnet-2019-02-26-14-40-33-819: Failed Reason: The primary container for production variant AllTraffic did not pass the ping health check.

My entry point code trasform.py looks like below where I am doing some preprocessing of image.

from __future__ import print_function

import argparse
import bisect
from collections import Counter
from itertools import chain, islice
import json
import logging
import time
import random
import os

import mxnet as mx
from mxnet import gluon, autograd, nd
from mxnet.io import DataIter, DataBatch, DataDesc
import numpy as np

from sagemaker_mxnet_container.training_utils import scheduler_host

import io
import boto3
from keras.preprocessing import image
from keras.applications.densenet import preprocess_input

logging.basicConfig(level=logging.DEBUG)



def transform_fn(net, data, input_content_type, output_content_type):
    """
    Transform a request using the Gluon model. Called once per request.
    :param net: The model.
    :param data: The request payload.
    :param input_content_type: The request content type.
    :param output_content_type: The (desired) response content type.
    :return: response payload and content type.
    """
    # we can use content types to vary input/output handling, but
    # here we just assume json for both
    net, vocab = net
    parsed = json.loads(data)
    outputs = []
   
    img_path = 'shoe.jpg'
    img = image.load_img(img_path, target_size=(256, 256))
    img_data = image.img_to_array(img)
    img_data = np.expand_dims(img_data, axis=0)
    img_data = preprocess_input(img_data)
    img_data_list = img_data.tolist()
    img_data_json = json.dumps(img_data_list)
    
    output = net(img_data_json)
    prediction = mx.nd.argmax(output, axis=1)
    outputs.append(int(prediction.asscalar()))
    response_body = json.dumps(outputs)
    return response_body, output_content_type

Minimal repro / logs

Error
ValueError: Error hosting endpoint sagemaker-mxnet-2019-02-26-14-40-33-819: Failed Reason: The primary container for production variant AllTraffic did not pass the ping health check.

• Exact command to reproduce:

[mvsusp]

Hi @nahidalam,

When the endpoint starts, SageMaker sends it ping requests to ensure that it started properly.

You can see the endpoint errors in cloudwatch logs. One possible way to find the logs is going to SageMaker AWS Console -> Endpoints -> click on your endpoint name -> click on 'view logs'

Thanks for using SageMaker

[nahidalam]

@mvsusp So the issue is my entry point code (python file) imports Keras etc. which is not found while creating the endpoint. Below is a cloudwatch error snippet

ImportError: No module named keras

04:15:21
[2019-02-27 04:15:21 +0000] [34] [INFO] Worker exiting (pid: 34)

04:15:21
[2019-02-27 04:15:21 +0000] [36] [ERROR] Exception in worker process

04:15:21
Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker worker.init_process() File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 203, in init_process super(GeventWorker, self).init_process() File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 129, in init_proc

I am a bit surprised as the notebook runs on conda_mxnet_p36 environment but the entry point python file can't access that environment. Why? Do I need to add something in the entry point python code.

Below is my entry point code

import argparse
import bisect
from collections import Counter
from itertools import chain, islice
import json
import logging
import time
import random
import os

import mxnet as mx
from mxnet import gluon, autograd, nd
from mxnet.io import DataIter, DataBatch, DataDesc
import numpy as np

import io
import boto3
import keras
from keras.preprocessing import image
from keras.applications.densenet import preprocess_input

logging.basicConfig(level=logging.DEBUG)

def transform_fn(net, data, input_content_type, output_content_type):
    """
    Transform a request using the Gluon model. Called once per request.
    :param net: The model.
    :param data: The request payload.
    :param input_content_type: The request content type.
    :param output_content_type: The (desired) response content type.
    :return: response payload and content type.
    """
    # we can use content types to vary input/output handling, but
    # here we just assume json for both
    net, vocab = net
    parsed = json.loads(data)
    outputs = []

    img_path = 'shoe.jpg'
    img = image.load_img(img_path, target_size=(256, 256))
    img_data = image.img_to_array(img)
    img_data = np.expand_dims(img_data, axis=0)
    img_data = preprocess_input(img_data)
    img_data_list = img_data.tolist()
    img_data_json = json.dumps(img_data_list)

    #is this the right way? input format to nda might not be right
    output = net(img_data_json)
    prediction = mx.nd.argmax(output, axis=1)
    outputs.append(int(prediction.asscalar()))
    response_body = json.dumps(outputs)
    return response_body, output_content_type

[iquintero]

Hi @nahidalam

Let me clarify, the conda environment (conda_mxnet_p36 in your case) is only used in your notebook instance. When you submit a training job, batch transform job or host an endpoint in sagemaker, you are not using that environment but instead it is using our Sagemaker MXNet docker container.

To fix this issue, please add a
requirements.txt file in your source_dir along with your python files.

You can also refer to our docs: https://sagemaker.readthedocs.io/en/stable/using_mxnet.html#using-third-party-libraries

You can use the python docs for a reference on the requirements.txt file syntax
https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format

Let me know if this works for you.

[nahidalam]

OK so my requirements.txt file now looks like this

keras==2.2.4
pillow==5.4.1

I did a pip install requirements.txt

But I am still getting error as it is not finding keras

[iquintero]

Hi @nahidalam

Sorry for the late reply. I took another look at your script and it looks like you are using keras.preprocessing and keras.applications. So your requirements.txt needs to have that listed as well.

For example

Keras-Preprocessing==1.0.9
Keras-Applications==1.0.7

Also remember the requirements.txt needs to be part of your .tar.gz file which contains transform_script.py

Also I noticed you have a question in the AWS Forums, Im going to point that question to this issue so that we can troubleshoot in a single central location.

[nahidalam]

@iquintero My model.tar.gz file contains only the model artifacts. Do you mean it should also contain my entry_point.py and requirements.txt?

[mvsusp]

Hi @nahidalam,

model.tar.gz only contains your model weights. Please ensure that your entrypoint file and your requirements.txt file are located under the same folder. For more information, please refer to the docs:
https://sagemaker.readthedocs.io/en/stable/using_mxnet.html#deploying-mxnet-models and https://sagemaker.readthedocs.io/en/stable/using_mxnet.html#using-third-party-libraries

[icywang86rui]

@nahidalam

Are you unblocked?

[icywang86rui]

Closing the issue for now. Feel free to reopen if you have any further questions.

[velociraptor111]

Hi, @mvsusp and @iquintero
Sorry for reopening an old issue. After reading all the posts above & the doc links above I am still unclear about the steps.

I'm trying to achieve the same thing whereas I am importing GluonCV in my file where I call
from gluoncv import data in my entry_point.py

I am confused on where to put the requirements.txt and if I have to do something like
pip install requirements.txt in my entry_point.py.

Here is how I am declaring MXNetModel in my jupyter notebook script:

sagemaker_model = MXNetModel(model_data = 's3://' + sagemaker_session.default_bucket() + '/model/yolo_object_person_detector.tar.gz',
                            role = role, 
                             entry_point = 'entry_point.py',
                             py_version='py3',
                             framework_version='1.4.1',
                            sagemaker_session = sagemaker_session)


transformer = sagemaker_model.transformer(instance_count=1, instance_type='ml.m4.xlarge', output_path=batch_output)

transformer.transform(data=batch_input, content_type='application/x-image')

I have tried putting a requirements.txt in the same directory of the entry_point.py and running the script above, but I'm still getting the error where it's unable to find GluonCV.

[Iam-ahsan]

Hi, I am deploying Resent-50 on Sagemaker. It works perfectly fine when I deploy up to 4 instances per endpoint. As soon as I try to add more nodes to the endpoint I get the following error.
"Failure reason
Request to service failed. If failure persists after retry, contact customer support."

Upon checking the logs for the endpoint I get the following error.
Traceback (most recent call last):
File "/usr/local/bin/dockerd-entrypoint.py", line 8, in
serving.main()
File "/usr/local/lib/python3.6/site-packages/sagemaker_mxnet_serving_container/serving.py", line 42, in main
model_server.start_model_server(handler_service=HANDLER_SERVICE)
File "/usr/local/lib/python3.6/site-packages/sagemaker_inference/model_server.py", line 63, in start_model_server
'/dev/null'])
File "/usr/local/lib/python3.6/subprocess.py", line 287, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/local/lib/python3.6/subprocess.py", line 729, in init
restore_signals, start_new_session)
File "/usr/local/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)

I double-checked my instance count limit and that's not the issue. I would really appreciate it if you guys could give me some pointer on how can I resolve this error.

[lu-liu-rft]

Hi, I am deploying Resent-50 on Sagemaker. It works perfectly fine when I deploy up to 4 instances per endpoint. As soon as I try to add more nodes to the endpoint I get the following error.
"Failure reason
Request to service failed. If failure persists after retry, contact customer support."

Upon checking the logs for the endpoint I get the following error.
Traceback (most recent call last):
File "/usr/local/bin/dockerd-entrypoint.py", line 8, in
serving.main()
File "/usr/local/lib/python3.6/site-packages/sagemaker_mxnet_serving_container/serving.py", line 42, in main
model_server.start_model_server(handler_service=HANDLER_SERVICE)
File "/usr/local/lib/python3.6/site-packages/sagemaker_inference/model_server.py", line 63, in start_model_server
'/dev/null'])
File "/usr/local/lib/python3.6/subprocess.py", line 287, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/local/lib/python3.6/subprocess.py", line 729, in init
restore_signals, start_new_session)
File "/usr/local/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)

I double-checked my instance count limit and that's not the issue. I would really appreciate it if you guys could give me some pointer on how can I resolve this error.

I'm experiencing a similar issue, might someone be able to help please?