From 9659e05cf6501369f9b3dc6e3f56ca110bb60893 Mon Sep 17 00:00:00 2001 From: eslesar-aws Date: Wed, 22 Apr 2020 14:55:54 -0700 Subject: [PATCH 1/7] doc: update PyTorch BYOM topic pytorch-byom-edits --- doc/using_pytorch.rst | 159 +++++++++++++++++++++++------------------- 1 file changed, 88 insertions(+), 71 deletions(-) diff --git a/doc/using_pytorch.rst b/doc/using_pytorch.rst index e33d2a2c9c..18462426b7 100644 --- a/doc/using_pytorch.rst +++ b/doc/using_pytorch.rst @@ -4,9 +4,9 @@ Using PyTorch with the SageMaker Python SDK With PyTorch Estimators and Models, you can train and host PyTorch models on Amazon SageMaker. -Supported versions of PyTorch: ``0.4.0``, ``1.0.0``, ``1.1.0``, ``1.2.0``, ``1.3.1``. +* Supported versions of PyTorch: ``0.4.0``, ``1.0.0``, ``1.1.0``, ``1.2.0``, ``1.3.1``. -Supported versions of PyTorch for Elastic Inference: ``1.3.1``. +* Supported versions of PyTorch for Elastic Inference: ``1.3.1``. We recommend that you use the latest supported version, because that's where we focus most of our development efforts. @@ -90,7 +90,7 @@ Note that SageMaker doesn't support argparse actions. If you want to use, for ex you need to specify `type` as `bool` in your script and provide an explicit `True` or `False` value for this hyperparameter when instantiating PyTorch Estimator. -For more on training environment variables, please visit `SageMaker Containers `_. +For more on training environment variables, see `SageMaker Containers `_. Save the Model -------------- @@ -115,7 +115,7 @@ to a certain filesystem path called ``model_dir``. This value is accessible thro with open(os.path.join(args.model_dir, 'model.pth'), 'wb') as f: torch.save(model.state_dict(), f) -After your training job is complete, SageMaker will compress and upload the serialized model to S3, and your model data +After your training job is complete, SageMaker compresses and uploads the serialized model to S3, and your model data will be available in the S3 ``output_path`` you specified when you created the PyTorch Estimator. If you are using Elastic Inference, you must convert your models to the TorchScript format and use ``torch.jit.save`` to save the model. @@ -514,11 +514,91 @@ The function should return a byte array of data serialized to content_type. The default implementation expects ``prediction`` to be a torch.Tensor and can serialize the result to JSON, CSV, or NPY. It accepts response content types of "application/json", "text/csv", and "application/x-npy". -Working with Existing Model Data and Training Jobs -================================================== -Attach to existing training jobs --------------------------------- +Bring your own model +==================== + +You can deploy a PyTorch model that you trained outside of SageMaker by using the ``PyTorchModel`` class. +Typically, you save a PyTorch model as a file with extension ``.pt`` or ``.pth``. +To do this, you need to: + +* Write an inference script. +* Package the model artifacts into a tar.gz file. +* Upload the tar.gz file to an S3 bucket. +* Create the ``PyTorchModel`` object. + +Write an inference script +------------------------- + +You must create an inference script that implements (at least) the ``predict_fn`` function that calls the loaded model to get a prediction. +Optionally, you can also implement ``input_fn`` and ``output_fn`` to process input and output. +For information about how to write an inference script, see `Serve a PyTorch Model <#serve-a-pytorch-model>`_. +Save the inference script as ``inference.py`` in the same folder where you saved your PyTorch model. + +Package model artifacts into a tar.gz file +------------------------------------------ + +The directory structure where you saved your PyTorch model should look something like the following: + +| my_model +| |--model.pth +| +| code +| |--inference.py +| |--requirements.txt + +Where ``requirments.txt`` is an optional file that specifies dependencies on third-party libraries. + +With this file structure, run the following command to package your model as a ``tar.gz`` file: + +``tar -czf model.tar.gz my_model code`` + +Upload model.tar.gz to S3 +------------------------- + +After you package your model into a ``tar.gz`` file, upload it to an S3 bucket by running the following python code: + +.. code:: python + + import boto3 + import sagemaker + s3 = boto3.client('s3') + + from sagemaker import get_execution_role + role = get_execution_role() + + response = s3.upload_file('model.tar.gz', 'my-bucket', '%s/%s' %('my-path', 'model.tar.gz')) + +Where ``my-bucket`` is the name of your S3 bucket, and ``my-path`` is the folder where you want to store the model. + + + You can also upload to S3 by using the AWS CLI: + + .. code:: bash + + aws s3 cp model.tar.gz s3://my-bucket/my-path/model.tar.gz + + +To run this command, you'll need to have the AWS CLI tool installed. For information about installing the AWS CLI, +see `Installing the AWS CLI `_. + +Create a PyTorchModel object +---------------------------- + +Now call the :class:`sagemaker.pytorch.model.PyTorchModel` constructor to create a model object, and then call its ``deploy()`` method to deploy your model for inference. + +.. code:: python + + pytorch_model = PyTorchModel(model_data='s3://my-bucket/my-path/model.tar.gz', role=role, + entry_point='inference.py') + + predictor = pytorch_model.deploy(instance_type='ml.c4.xlarge', initial_instance_count=1) + + +Now you can call the ``predict()`` method to get predictions from your deployed model. + +Attach an estimator to an existing training job +=============================================== You can attach a PyTorch Estimator to an existing training job using the ``attach`` method. @@ -540,69 +620,6 @@ The ``attach`` method accepts the following arguments: - ``sagemaker_session:`` The Session used to interact with SageMaker -Deploy Endpoints from model data --------------------------------- - -In addition to attaching to existing training jobs, you can deploy models directly from model data in S3. -The following code sample shows how to do this, using the ``PyTorchModel`` class. - -.. code:: python - - pytorch_model = PyTorchModel(model_data='s3://bucket/model.tar.gz', role='SageMakerRole', - entry_point='transform_script.py') - - predictor = pytorch_model.deploy(instance_type='ml.c4.xlarge', initial_instance_count=1) - -The PyTorchModel constructor takes the following arguments: - -- ``model_dat:`` An S3 location of a SageMaker model data - .tar.gz file -- ``image:`` A Docker image URI -- ``role:`` An IAM role name or Arn for SageMaker to access AWS - resources on your behalf. -- ``predictor_cls:`` A function to - call to create a predictor. If not None, ``deploy`` will return the - result of invoking this function on the created endpoint name -- ``env:`` Environment variables to run with - ``image`` when hosted in SageMaker. -- ``name:`` The model name. If None, a default model name will be - selected on each ``deploy.`` -- ``entry_point:`` Path (absolute or relative) to the Python file - which should be executed as the entry point to model hosting. -- ``source_dir:`` Optional. Path (absolute or relative) to a - directory with any other training source code dependencies including - the entry point file. Structure within this directory will be - preserved when training on SageMaker. -- ``enable_cloudwatch_metrics:`` Optional. If true, training - and hosting containers will generate Cloudwatch metrics under the - AWS/SageMakerContainer namespace. -- ``container_log_level:`` Log level to use within the container. - Valid values are defined in the Python logging module. -- ``code_location:`` Optional. Name of the S3 bucket where your - custom code will be uploaded to. If not specified, will use the - SageMaker default bucket created by sagemaker.Session. -- ``sagemaker_session:`` The SageMaker Session - object, used for SageMaker interaction - -Your model data must be a .tar.gz file in S3. SageMaker Training Job model data is saved to .tar.gz files in S3, -however if you have local data you want to deploy, you can prepare the data yourself. - -Assuming you have a local directory containg your model data named "my_model" you can tar and gzip compress the file and -upload to S3 using the following commands: - -:: - - tar -czf model.tar.gz my_model - aws s3 cp model.tar.gz s3://my-bucket/my-path/model.tar.gz - -This uploads the contents of my_model to a gzip compressed tar file to S3 in the bucket "my-bucket", with the key -"my-path/model.tar.gz". - -To run this command, you'll need the AWS CLI tool installed. Please refer to our `FAQ`_ for more information on -installing this. - -.. _FAQ: ../../../README.rst#faq - ************************* PyTorch Training Examples ************************* From 7c4dfa20b56689f319b31f35a67d4cd9cfab7653 Mon Sep 17 00:00:00 2001 From: eslesar-aws Date: Tue, 5 May 2020 10:52:30 -0700 Subject: [PATCH 2/7] doc: fix sphinx issues in using_pytorch.rst --- doc/using_pytorch.rst | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/doc/using_pytorch.rst b/doc/using_pytorch.rst index 3770d69cf7..76a98b93e2 100644 --- a/doc/using_pytorch.rst +++ b/doc/using_pytorch.rst @@ -4,11 +4,7 @@ Using PyTorch with the SageMaker Python SDK With PyTorch Estimators and Models, you can train and host PyTorch models on Amazon SageMaker. -<<<<<<< HEAD -* Supported versions of PyTorch: ``0.4.0``, ``1.0.0``, ``1.1.0``, ``1.2.0``, ``1.3.1``. -======= Supported versions of PyTorch: ``0.4.0``, ``1.0.0``, ``1.1.0``, ``1.2.0``, ``1.3.1``, ``1.4.0``. ->>>>>>> 53fe1dc2025a1ba6e7fe4f16f120dfcc245ed465 * Supported versions of PyTorch for Elastic Inference: ``1.3.1``. @@ -596,12 +592,14 @@ Package model artifacts into a tar.gz file The directory structure where you saved your PyTorch model should look something like the following: -| my_model -| |--model.pth -| -| code -| |--inference.py -| |--requirements.txt +:: + + | my_model + | |--model.pth + | + | code + | |--inference.py + | |--requirements.txt Where ``requirments.txt`` is an optional file that specifies dependencies on third-party libraries. From db84ea7365f464fda1019cccf14ce0425483a3e6 Mon Sep 17 00:00:00 2001 From: eslesar-aws Date: Tue, 5 May 2020 11:16:06 -0700 Subject: [PATCH 3/7] doc: fix sphinx issues in using_pytortch.rst --- doc/using_pytorch.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/using_pytorch.rst b/doc/using_pytorch.rst index 76a98b93e2..9629f328ce 100644 --- a/doc/using_pytorch.rst +++ b/doc/using_pytorch.rst @@ -583,9 +583,9 @@ Write an inference script ------------------------- You must create an inference script that implements (at least) the ``predict_fn`` function that calls the loaded model to get a prediction. -Optionally, you can also implement ``input_fn`` and ``output_fn`` to process input and output. +Optionally, you can also implement ``input_fn`` and ``output_fn`` to process input and output. For information about how to write an inference script, see `Serve a PyTorch Model <#serve-a-pytorch-model>`_. -Save the inference script as ``inference.py`` in the same folder where you saved your PyTorch model. +Save the inference script as ``inference.py`` in the same folder where you saved your PyTorch model. Package model artifacts into a tar.gz file ------------------------------------------ @@ -596,7 +596,7 @@ The directory structure where you saved your PyTorch model should look something | my_model | |--model.pth - | + | | code | |--inference.py | |--requirements.txt From e2cf89cd611e35410c199405a16d67e8d4045934 Mon Sep 17 00:00:00 2001 From: Eric Slesar <34587362+eslesar-aws@users.noreply.github.com> Date: Thu, 7 May 2020 10:39:22 -0700 Subject: [PATCH 4/7] Apply suggestions from code review doc: apply review suggestions from @ajaykarpur Co-authored-by: Ajay Karpur --- doc/using_pytorch.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/using_pytorch.rst b/doc/using_pytorch.rst index 9629f328ce..cc5418875f 100644 --- a/doc/using_pytorch.rst +++ b/doc/using_pytorch.rst @@ -90,7 +90,7 @@ Note that SageMaker doesn't support argparse actions. If you want to use, for ex you need to specify `type` as `bool` in your script and provide an explicit `True` or `False` value for this hyperparameter when instantiating PyTorch Estimator. -For more on training environment variables, see `SageMaker Containers `_. +For more on training environment variables, see the `SageMaker Training Toolkit `_. Save the Model -------------- @@ -575,8 +575,8 @@ Typically, you save a PyTorch model as a file with extension ``.pt`` or ``.pth`` To do this, you need to: * Write an inference script. -* Package the model artifacts into a tar.gz file. -* Upload the tar.gz file to an S3 bucket. +* Package the model artifacts into a ``tar.gz`` file. +* Upload the ``tar.gz`` file to an S3 bucket. * Create the ``PyTorchModel`` object. Write an inference script From ad29c6c10e4efe9f071f535ef65b3b9060aa6d32 Mon Sep 17 00:00:00 2001 From: eslesar-aws Date: Fri, 8 May 2020 16:33:02 -0700 Subject: [PATCH 5/7] doc: address review comments --- doc/using_pytorch.rst | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/doc/using_pytorch.rst b/doc/using_pytorch.rst index 4c7e95eb86..a89f852837 100644 --- a/doc/using_pytorch.rst +++ b/doc/using_pytorch.rst @@ -582,7 +582,10 @@ To do this, you need to: Write an inference script ------------------------- -You must create an inference script that implements (at least) the ``predict_fn`` function that calls the loaded model to get a prediction. +You must create an inference script that implements (at least) the ``model_fn`` function that calls the loaded model to get a prediction. + +**Note**: If you use elastic inference with PyTorch, you can use the default ``model_fn`` implementation provided in the serving container. + Optionally, you can also implement ``input_fn`` and ``output_fn`` to process input and output. For information about how to write an inference script, see `Serve a PyTorch Model <#serve-a-pytorch-model>`_. Save the inference script as ``inference.py`` in the same folder where you saved your PyTorch model. @@ -592,6 +595,9 @@ Package model artifacts into a tar.gz file The directory structure where you saved your PyTorch model should look something like the following: +**Note:** This directory struture is for PyTorch versions 1.2 and higher. For the directory structure for versions 1.1 and lower, +see `For versions 1.1 and lower <#for-versions-1.1-and-lower>`_. + :: | my_model @@ -610,7 +616,7 @@ With this file structure, run the following command to package your model as a ` Upload model.tar.gz to S3 ------------------------- -After you package your model into a ``tar.gz`` file, upload it to an S3 bucket by running the following python code: +After you package your model into a ``tar.gz`` file, upload it to an S3 bucket by running the following Python code: .. code:: python @@ -626,18 +632,18 @@ After you package your model into a ``tar.gz`` file, upload it to an S3 bucket b Where ``my-bucket`` is the name of your S3 bucket, and ``my-path`` is the folder where you want to store the model. - You can also upload to S3 by using the AWS CLI: +You can also upload to S3 by using the AWS CLI: - .. code:: bash - - aws s3 cp model.tar.gz s3://my-bucket/my-path/model.tar.gz +.. code:: python + + aws s3 cp model.tar.gz s3://my-bucket/my-path/model.tar.gz`` To run this command, you'll need to have the AWS CLI tool installed. For information about installing the AWS CLI, see `Installing the AWS CLI `_. -Create a PyTorchModel object ----------------------------- +Create a ``PyTorchModel`` object +-------------------------------- Now call the :class:`sagemaker.pytorch.model.PyTorchModel` constructor to create a model object, and then call its ``deploy()`` method to deploy your model for inference. @@ -651,8 +657,9 @@ Now call the :class:`sagemaker.pytorch.model.PyTorchModel` constructor to create Now you can call the ``predict()`` method to get predictions from your deployed model. +*********************************************** Attach an estimator to an existing training job -=============================================== +*********************************************** You can attach a PyTorch Estimator to an existing training job using the ``attach`` method. From 20b21bca0bd1dc53cd8631b8edad5cb297bcdd6b Mon Sep 17 00:00:00 2001 From: eslesar-aws Date: Mon, 11 May 2020 16:14:34 -0700 Subject: [PATCH 6/7] doc: address feedback in using_pytorch.rst --- doc/using_pytorch.rst | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/doc/using_pytorch.rst b/doc/using_pytorch.rst index a89f852837..76659d13d8 100644 --- a/doc/using_pytorch.rst +++ b/doc/using_pytorch.rst @@ -6,7 +6,7 @@ With PyTorch Estimators and Models, you can train and host PyTorch models on Ama Supported versions of PyTorch: ``0.4.0``, ``1.0.0``, ``1.1.0``, ``1.2.0``, ``1.3.1``, ``1.4.0``, ``1.5.0``. -* Supported versions of PyTorch for Elastic Inference: ``1.3.1``. +Supported versions of PyTorch for Elastic Inference: ``1.3.1``. We recommend that you use the latest supported version because that's where we focus our development efforts. @@ -586,9 +586,11 @@ You must create an inference script that implements (at least) the ``model_fn`` **Note**: If you use elastic inference with PyTorch, you can use the default ``model_fn`` implementation provided in the serving container. -Optionally, you can also implement ``input_fn`` and ``output_fn`` to process input and output. +Optionally, you can also implement ``input_fn`` and ``output_fn`` to process input and output, +and ``predict_fn`` to customize how the model server gets predictions from the loaded model. For information about how to write an inference script, see `Serve a PyTorch Model <#serve-a-pytorch-model>`_. -Save the inference script as ``inference.py`` in the same folder where you saved your PyTorch model. +Save the inference script in the same folder where you saved your PyTorch model. +Pass the filename of the inference script as the ``entry_point`` parameter when you create the ``PyTorchModel`` object. Package model artifacts into a tar.gz file ------------------------------------------ @@ -616,6 +618,10 @@ With this file structure, run the following command to package your model as a ` Upload model.tar.gz to S3 ------------------------- +**Note**: This step is optional. The ``PyTorchModel`` object will upload your model file to S3. +You can specify a bucket location by providing a value for the ``code_location`` parameter when you create the ``PyTorchModel`` object, +or you can use the default S3 session bucket. + After you package your model into a ``tar.gz`` file, upload it to an S3 bucket by running the following Python code: .. code:: python @@ -624,10 +630,8 @@ After you package your model into a ``tar.gz`` file, upload it to an S3 bucket b import sagemaker s3 = boto3.client('s3') - from sagemaker import get_execution_role - role = get_execution_role() - - response = s3.upload_file('model.tar.gz', 'my-bucket', '%s/%s' %('my-path', 'model.tar.gz')) + from sagemaker.s3 import S3Uploader + S3Uploader.upload('model.tar.gz', s3:://my-bucket/my-path/model.tar.gz') Where ``my-bucket`` is the name of your S3 bucket, and ``my-path`` is the folder where you want to store the model. @@ -636,7 +640,7 @@ You can also upload to S3 by using the AWS CLI: .. code:: python - aws s3 cp model.tar.gz s3://my-bucket/my-path/model.tar.gz`` + aws s3 cp model.tar.gz s3://my-bucket/my-path/model.tar.gz To run this command, you'll need to have the AWS CLI tool installed. For information about installing the AWS CLI, @@ -649,6 +653,9 @@ Now call the :class:`sagemaker.pytorch.model.PyTorchModel` constructor to create .. code:: python + from sagemaker import get_execution_role + role = get_execution_role() + pytorch_model = PyTorchModel(model_data='s3://my-bucket/my-path/model.tar.gz', role=role, entry_point='inference.py') From 8ea4170556bd9868420118704a7a98b822d8994c Mon Sep 17 00:00:00 2001 From: eslesar-aws Date: Thu, 14 May 2020 15:25:08 -0700 Subject: [PATCH 7/7] doc: fix trailing space errors and address feedback --- doc/using_pytorch.rst | 50 ++++++++----------------------------------- 1 file changed, 9 insertions(+), 41 deletions(-) diff --git a/doc/using_pytorch.rst b/doc/using_pytorch.rst index 76659d13d8..995e030067 100644 --- a/doc/using_pytorch.rst +++ b/doc/using_pytorch.rst @@ -575,8 +575,7 @@ Typically, you save a PyTorch model as a file with extension ``.pt`` or ``.pth`` To do this, you need to: * Write an inference script. -* Package the model artifacts into a ``tar.gz`` file. -* Upload the ``tar.gz`` file to an S3 bucket. +* Create the directory structure for your model files. * Create the ``PyTorchModel`` object. Write an inference script @@ -592,12 +591,16 @@ For information about how to write an inference script, see `Serve a PyTorch Mod Save the inference script in the same folder where you saved your PyTorch model. Pass the filename of the inference script as the ``entry_point`` parameter when you create the ``PyTorchModel`` object. -Package model artifacts into a tar.gz file ------------------------------------------- +Create the directory structure for your model files +--------------------------------------------------- + +You have to create a directory structure and place your model files in the correct location. +The ``PyTorchModel`` constructor packs the files into a ``tar.gz`` file and uploads it to S3. The directory structure where you saved your PyTorch model should look something like the following: -**Note:** This directory struture is for PyTorch versions 1.2 and higher. For the directory structure for versions 1.1 and lower, +**Note:** This directory struture is for PyTorch versions 1.2 and higher. +For the directory structure for versions 1.1 and lower, see `For versions 1.1 and lower <#for-versions-1.1-and-lower>`_. :: @@ -611,41 +614,6 @@ see `For versions 1.1 and lower <#for-versions-1.1-and-lower>`_. Where ``requirments.txt`` is an optional file that specifies dependencies on third-party libraries. -With this file structure, run the following command to package your model as a ``tar.gz`` file: - -``tar -czf model.tar.gz my_model code`` - -Upload model.tar.gz to S3 -------------------------- - -**Note**: This step is optional. The ``PyTorchModel`` object will upload your model file to S3. -You can specify a bucket location by providing a value for the ``code_location`` parameter when you create the ``PyTorchModel`` object, -or you can use the default S3 session bucket. - -After you package your model into a ``tar.gz`` file, upload it to an S3 bucket by running the following Python code: - -.. code:: python - - import boto3 - import sagemaker - s3 = boto3.client('s3') - - from sagemaker.s3 import S3Uploader - S3Uploader.upload('model.tar.gz', s3:://my-bucket/my-path/model.tar.gz') - -Where ``my-bucket`` is the name of your S3 bucket, and ``my-path`` is the folder where you want to store the model. - - -You can also upload to S3 by using the AWS CLI: - -.. code:: python - - aws s3 cp model.tar.gz s3://my-bucket/my-path/model.tar.gz - - -To run this command, you'll need to have the AWS CLI tool installed. For information about installing the AWS CLI, -see `Installing the AWS CLI `_. - Create a ``PyTorchModel`` object -------------------------------- @@ -655,7 +623,7 @@ Now call the :class:`sagemaker.pytorch.model.PyTorchModel` constructor to create from sagemaker import get_execution_role role = get_execution_role() - + pytorch_model = PyTorchModel(model_data='s3://my-bucket/my-path/model.tar.gz', role=role, entry_point='inference.py')