feat: Integ tests for jumpstart model and estimator #2865

evakravi · 2022-01-24T16:09:14Z

Issue #, if available:

Description of changes:
This PR adds integration test support for script mode usage of the Model and Estimator classes.
It also fixes a bug in the Model class that prevented the repackaged artifacts from being referenced.

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-01-24T16:29:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 56b3d71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-01-24T16:59:20Z

Codecov Report

Merging #2865 (fe94df3) into master-jumpstart (63b0372) will not change coverage.
The diff coverage is n/a.

@@                Coverage Diff                @@
##           master-jumpstart    #2865   +/-   ##
=================================================
  Coverage             89.27%   89.27%           
=================================================
  Files                   186      186           
  Lines                 16063    16063           
=================================================
  Hits                  14341    14341           
  Misses                 1722     1722

Impacted Files	Coverage Δ
src/sagemaker/model.py	`90.12% <ø> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 63b0372...fe94df3. Read the comment docs.

sagemaker-bot · 2022-01-24T17:13:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 56b3d71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

tests/integ/sagemaker/jumpstart/script_mode_class/test_transfer_learning.py

JGuinegagne · 2022-01-24T17:35:09Z

tests/integ/sagemaker/jumpstart/utils.py

+class EndpointInvoker:
+    def __init__(
+        self,
+        endpoint_name,


nit: endpoint_name: str

sagemaker-bot · 2022-01-24T17:48:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 56b3d71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…cript-model-estimator

sagemaker-bot · 2022-01-24T19:15:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 59defdb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T19:20:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e93e001
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T19:57:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 59defdb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T19:57:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e93e001
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T20:10:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f8f7d4b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T20:33:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: e93e001
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T20:33:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 59defdb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T20:35:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 59defdb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T20:49:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: e93e001
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T20:51:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f8f7d4b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T21:13:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: fe94df3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T21:34:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: f8f7d4b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T21:40:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: f8f7d4b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T21:54:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: fe94df3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-24T22:34:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: fe94df3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2022-01-25T18:00:31Z

tests/integ/sagemaker/jumpstart/script_mode_class/test_inference.py

+
+    response = endpoint_invoker.invoke_tabular_endpoint(features)
+
+    assert response is not None


nit: can we assert response to be of some specific value? for better validation

Ideally, yes. But we may change the model input/output format, and would like to make these tests as low maintenance as possible. We have separate tests to verify the integrity of individual models.

feat: integ tests for jumpstart model and estimator

56b3d71

JGuinegagne reviewed Jan 24, 2022

View reviewed changes

evakravi and others added 2 commits January 24, 2022 18:56

change: resolve git comments

59defdb

Merge branch 'master-jumpstart' into feat/integ-tests-for-jumpstart-s…

e93e001

…cript-model-estimator

change: use model_uri estimator class attribute in jumpstart integ tests

f8f7d4b

JGuinegagne approved these changes Jan 24, 2022

View reviewed changes

change: make training jumpstart test more generic

fe94df3

mufaddal-rohawala approved these changes Jan 25, 2022

View reviewed changes

shreyapandit approved these changes Jan 25, 2022

View reviewed changes

shreyapandit changed the title ~~feat: integ tests for jumpstart model and estimator~~ feat: Integ tests for jumpstart model and estimator Jan 25, 2022

shreyapandit merged commit c9aa29b into aws:master-jumpstart Jan 25, 2022


		response = endpoint_invoker.invoke_tabular_endpoint(features)

		assert response is not None

feat: Integ tests for jumpstart model and estimator #2865

feat: Integ tests for jumpstart model and estimator #2865

Uh oh!

Conversation

evakravi commented Jan 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

codecov-commenter commented Jan 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

Uh oh!

JGuinegagne Jan 24, 2022

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jan 24, 2022

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala Jan 25, 2022

Choose a reason for hiding this comment

Uh oh!

evakravi Jan 25, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

evakravi commented Jan 24, 2022 •

edited

Loading

codecov-commenter commented Jan 24, 2022 •

edited

Loading