Merge branch 'master' of https://github.com/aws/sagemaker-python-sdk into refactor-mxnet

Author: eslesar-aws

.pylintrc

@@ -83,7 +83,6 @@ disable=
     too-many-arguments,
     invalid-name,
     too-many-instance-attributes,
-    line-too-long, # We let Flake8 take care of this # TODO: Fix these and stop relying on flake8
     len-as-condition, # TODO: Enable this check once pylint 2.4.0 is released and consumed due to the fix in https://github.com/PyCQA/pylint/issues/2684
     import-error, # Since we run Pylint before any of our builds in tox, this will always fail
     protected-access, # TODO: Fix access
@@ -228,8 +227,9 @@ max-nested-blocks=5
 # Maximum number of characters on a single line.
 max-line-length=100
 
-# Regexp for a line that is allowed to be longer than the limit.
-ignore-long-lines=^\s*(# )?<?https?://\S+>?$
+# Regexp for a line that is allowed to be longer than the limit. Can only be a single regex.
+# The following matches any semblance of a url of any sort.
+ignore-long-lines=^\s*.*(# )?(<?https?:\/\/)?[-a-zA-Z0-9@:%%._\+~#=><]{1,256}\.[a-zA-Z0-9()]{1,6}\b[-a-zA-Z0-9()@:%%_\+.~#?&\/\/=<>]*\S*>?$
 
 # Allow the body of an if to be on the same line as the test if there is no
 # else.

CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## v1.34.1 (2019-07-23)
+
+### Bug fixes and other changes
+
+ * enable line-too-long Pylint check
+ * improving Chainer integ tests
+ * update TensorFlow script mode dependency list
+ * improve documentation of some functions
+ * update PyTorch version
+ * allow serving script to be defined for deploy() and transformer() with frameworks
+ * format and add missing docstring placeholders
+ * add MXNet 1.4.1 support
+
+### Documentation changes
+
+ * add instructions for setting up Cloud9 environment.
+ * update using_tensorflow topic
+
 ## v1.34.0 (2019-07-18)
 
 ### Features

CONTRIBUTING.md

@@ -19,6 +19,23 @@ reported the issue. Please try to include as much information as you can. Detail
 * Any modifications you've made relevant to the bug
 * A description of your environment or deployment
 
+## Setting up your development environment [optional, but recommended]
+
+* Set up the Cloud9 environment:
+  * Instance type: You'll need at least 4 GB of RAM to avoid running into memory issues. We recommend at least a t3.medium to run the unit tests. Larger hosts will reduce the chance of encountering resource limits.
+  * Follow the instructions at [Creating a Cloud9 EC2 Environment](https://docs.aws.amazon.com/cloud9/latest/user-guide/create-environment.html#create-environment-main) to set up a Cloud9 EC2 environment
+* Expand the storage of the EC2 instance from 10GB to 20GB
+  * Because you'll need a minimum of 11GB of disk storage on the EC2 instance to run the repository's unit tests, you'll need to expand your EC2 volume size. We recommend at least 20GB. A larger volume will reduce the chance of encountering resource limits. 
+  * Follow the instructions at [Modifying an EBS Volume Using Elastic Volumes (Console)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/requesting-ebs-volume-modifications.html#modify-ebs-volume) to increase the EBS volume size associated with the newly created EC2 instance.
+  * Wait 5-10min for the new EBS volume increase to take effect.
+  * Allow EC2 to claim the additional space by stopping and then starting your EC2 host.
+* Create a fork of this repository on GitHub. You should end up with a fork at `https://github.com/<username>/sagemaker-python-sdk`
+  * Follow the instructions at [Fork a repo](https://help.github.com/en/articles/fork-a-repo) to fork a GitHub repository.
+* In the Cloud9 UI, pull down this repository by clicking on "Clone from Github" or running the following command in the Cloud9 terminal: `git clone https://github.com/<username>/sagemaker-python-sdk` where <username> is your github username.
+* Install tox using `pip install tox`
+* Install coverage using `pip install .[test]`
+* cd into the sagemaker-python-sdk folder: `cd sagemaker-python-sdk` or `cd /environment/sagemaker-python-sdk`
+* Run the following tox command and verify that all unit tests pass: `tox tests/unit`
 
 ## Contributing via Pull Requests
 Contributions via pull requests are much appreciated. 

README.rst

@@ -191,7 +191,7 @@ By using TensorFlow SageMaker Estimators, you can train and host TensorFlow mode
 
 Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``, ``1.6.0``, ``1.7.0``, ``1.8.0``, ``1.9.0``, ``1.10.0``, ``1.11.0``, ``1.12.0``, ``1.13.1``.
 
-Supported versions of TensorFlow for Elastic Inference: ``1.11.0``, ``1.12.0``.
+Supported versions of TensorFlow for Elastic Inference: ``1.11.0``, ``1.12.0``, ``1.13.0``
 
 We recommend that you use the latest supported version, because that's where we focus most of our development efforts.
 

VERSION

@@ -1 +1 @@
-1.34.1.dev0
+1.34.2.dev0

src/sagemaker/algorithm.py

@@ -128,7 +128,7 @@ def __init__(
                     (default: None). If specified, the estimator will create a channel pointing to
                     the model so the training job can download it. This model
                     can be a 'model.tar.gz' from a previous training job, or
-                    other artifacts coming from a different source. More
+                    other artifacts coming from a different source.
                     More information:
                     https://docs.aws.amazon.com/sagemaker/latest/dg/cdf-training.html#td-deserialization
 

src/sagemaker/amazon/linear_learner.py

@@ -83,8 +83,8 @@ class LinearLearner(AmazonAlgorithmEstimatorBase):
             "softmax_loss",
             "auto",
         ),
-        '"logistic", "squared_loss", "absolute_loss", "hinge_loss", "eps_insensitive_squared_loss", '
-        '"eps_insensitive_absolute_loss", "quantile_loss", "huber_loss", "softmax_loss" or "auto"',
+        '"logistic", "squared_loss", "absolute_loss", "hinge_loss", "eps_insensitive_squared_loss",'
+        ' "eps_insensitive_absolute_loss", "quantile_loss", "huber_loss", "softmax_loss" or "auto"',
         str,
     )
     wd = hp("wd", ge(0), "A float greater-than or equal to 0", float)
@@ -369,7 +369,8 @@ def __init__(
             num_classes is None or num_classes < 3
         ):
             raise ValueError(
-                "For predictor_type 'multiclass_classifier', 'num_classes' should be set to a value greater than 2."
+                "For predictor_type 'multiclass_classifier', 'num_classes' should be set to a "
+                "value greater than 2."
             )
 
     def create_model(self, vpc_config_override=VPC_CONFIG_DEFAULT):

src/sagemaker/analytics.py

@@ -176,7 +176,7 @@ def description(self, force_refresh=False):
         if force_refresh:
             self.clear_cache()
         if not self._tuning_job_describe_result:
-            self._tuning_job_describe_result = self._sage_client.describe_hyper_parameter_tuning_job(
+            self._tuning_job_describe_result = self._sage_client.describe_hyper_parameter_tuning_job(  # noqa: E501 # pylint: disable=line-too-long
                 HyperParameterTuningJobName=self.name
             )
         return self._tuning_job_describe_result
@@ -288,10 +288,12 @@ def _determine_timeinterval(self):
         description = self._sage_client.describe_training_job(TrainingJobName=self.name)
         start_time = self._start_time or description[u"TrainingStartTime"]  # datetime object
         # Incrementing end time by 1 min since CloudWatch drops seconds before finding the logs.
-        # This results in logs being searched in the time range in which the correct log line was not present.
+        # This results in logs being searched in the time range in which the correct log line was
+        # not present.
         # Example - Log time - 2018-10-22 08:25:55
-        #           Here calculated end time would also be 2018-10-22 08:25:55 (without 1 min addition)
-        #           CW will consider end time as 2018-10-22 08:25 and will not be able to search the correct log.
+        #       Here calculated end time would also be 2018-10-22 08:25:55 (without 1 min addition)
+        #       CW will consider end time as 2018-10-22 08:25 and will not be able to search the
+        #           correct log.
         end_time = self._end_time or description.get(
             u"TrainingEndTime", datetime.datetime.utcnow()
         ) + datetime.timedelta(minutes=1)

src/sagemaker/chainer/estimator.py

@@ -177,8 +177,9 @@ def create_model(
                 the model. Default: use subnets and security groups from this Estimator.
                 * 'Subnets' (list[str]): List of subnet ids.
                 * 'SecurityGroupIds' (list[str]): List of security group ids.
-            entry_point (str): Path (absolute or relative) to the local Python source file which should be executed
-                as the entry point to training. If not specified, the training entry point is used.
+            entry_point (str): Path (absolute or relative) to the local Python source file which
+                should be executed as the entry point to training. If not specified, the training
+                entry point is used.
             source_dir (str): Path (absolute or relative) to a directory with any other serving
                 source code dependencies aside from the entry point file.
                 If not specified, the model source directory from training is used.

src/sagemaker/estimator.py

@@ -279,15 +279,13 @@ def fit(self, inputs=None, wait=True, logs=True, job_name=None):
 
                 * (str) the S3 location where training data is saved.
 
-                * (dict[str, str] or dict[str, sagemaker.session.s3_input]) If using multiple channels for
-                      training data, you can specify a dict mapping channel
-                      names to strings or :func:`~sagemaker.session.s3_input`
-                      objects.
-
-                * (sagemaker.session.s3_input) - channel configuration for S3 data sources that can provide
-                      additional information as well as the path to the training
-                      dataset. See :func:`sagemaker.session.s3_input` for full
-                      details.
+                * (dict[str, str] or dict[str, sagemaker.session.s3_input]) If using multiple
+                    channels for training data, you can specify a dict mapping channel names to
+                    strings or :func:`~sagemaker.session.s3_input` objects.
+
+                * (sagemaker.session.s3_input) - channel configuration for S3 data sources that can
+                    provide additional information as well as the path to the training dataset.
+                    See :func:`sagemaker.session.s3_input` for full details.
             wait (bool): Whether the call should wait until the job completes
                 (default: True).
             logs (bool): Whether to show the logs produced by the job. Only
@@ -990,9 +988,9 @@ def create_model(
     ):
         """Create a model to deploy.
 
-        The serializer, deserializer, content_type, and accept arguments are only used to define a default
-        RealTimePredictor. They are ignored if an explicit predictor class is passed in. Other arguments
-        are passed through to the Model class.
+        The serializer, deserializer, content_type, and accept arguments are only used to define a
+        default RealTimePredictor. They are ignored if an explicit predictor class is passed in.
+        Other arguments are passed through to the Model class.
 
         Args:
             role (str): The ``ExecutionRoleArn`` IAM Role ARN for the ``Model``,
@@ -1562,8 +1560,9 @@ def transformer(
                 worker per vCPU.
             volume_kms_key (str): Optional. KMS key ID for encrypting the volume
                 attached to the ML compute instance (default: None).
-            entry_point (str): Path (absolute or relative) to the local Python source file which should be executed
-                as the entry point to training. If not specified, the training entry point is used.
+            entry_point (str): Path (absolute or relative) to the local Python source file which
+                should be executed as the entry point to training. If not specified, the training
+                entry point is used.
 
         Returns:
             sagemaker.transformer.Transformer: a ``Transformer`` object that can be used to start a