change: expose num_clusters parameter for clarify shap in shapconfig #2713

prkrishnan1 · 2021-10-18T20:21:30Z

Description of changes:
Added an optional parameter 'num_clusters' to SHAPConfig class to allow users to specify the number of clusters used by K-Means clustering. This parameter is used when a baseline dataset is not provided for SHAP, so a dataset is generated using K-means clustering. Code ensures baseline and num_clusters parameters aren't both passed, since they are mutually exclusive.

Testing done:
Unit Testing

test_clarify::test_shap_config_no_baseline -> Validates that SHAPConfig object without baseline and with num_clusters can be created
test_clarify::test_shap_config_no_parameters -> Validates that SHAPConfig object without any provided parameters can be created
test_clarify::test_invalid_shap_config -> Validates that providing baseline and num_clusters parameters to SHAPConfig results in an error

End to End Testing
For documentation purposes, Clarify integration tests using below objects ran successfully.

SHAPConfig(num_samples=2, agg_method="mean_sq", seed=123)
SHAPConfig(num_samples=2, agg_method="mean_sq", seed=123, num_clusters=3)

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2021-10-18T20:26:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 426c230
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-18T21:08:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 426c230
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-18T21:09:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 426c230
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-18T21:25:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 0b03bc2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/clarify.py

sagemaker-bot · 2021-10-19T00:10:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 18c25b5
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-19T00:55:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 18c25b5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-19T00:57:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 18c25b5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-20T17:30:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 8eb0e57
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-20T17:43:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e87ad0b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-20T17:57:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b12677d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-20T18:10:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 8eb0e57
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-20T18:14:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 8eb0e57
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-20T18:25:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b12677d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-20T18:28:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: e87ad0b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-20T18:31:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e87ad0b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-20T18:31:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b12677d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

xgchena

I just realized that we don't have an integration test for AutoSHAP (no baseline) yet, could you add one to tests/integ/test_clarify.py? (You can refer to

sagemaker-python-sdk/tests/integ/test_clarify.py

Line 237 in 1f80314

    
           def test_shap(clarify_processor, data_config, model_config, shap_config, sagemaker_session):

, create a SHAPConfig with only num_clusters specified.)

src/sagemaker/clarify.py

tests/unit/test_clarify.py

src/sagemaker/clarify.py

sagemaker-bot · 2021-10-21T23:50:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 577a232
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

xgchena

The new revision looks good.

sagemaker-bot · 2021-10-22T21:57:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 907f5b4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

navinsoni · 2021-10-23T10:11:54Z

src/sagemaker/clarify.py

-
+        if num_clusters is not None and baseline is not None:
+            raise ValueError(
+                "Baseline and num_clusters cannot both be provided. Please specify one of the two."


nit: please change it to

baseline and num_clusters cannot be provided together. Please specify one of the two."

navinsoni · 2021-10-23T10:15:45Z

tests/unit/test_clarify.py

+    with pytest.raises(ValueError) as error:
+        SHAPConfig(baseline=[[1]], num_samples=1, agg_method="mean_abs", num_clusters=2)
+    assert (
+        "Baseline and num_clusters cannot both be provided. Please specify one of the two."


sagemaker-bot · 2021-10-25T19:13:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: ee34924
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-25T19:33:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: ed034ed
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-25T19:55:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: ee34924
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-25T20:02:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: ee34924
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-25T20:04:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c6c4c9d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-25T20:08:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: ed034ed
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-25T20:38:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c6c4c9d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-25T20:47:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c6c4c9d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

shreyapandit

LGTM

sagemaker-bot · 2021-10-26T17:54:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 7ee3425
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-26T18:26:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 7ee3425
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-26T18:31:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 7ee3425
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-26T21:43:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 435f8cd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-26T22:25:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 435f8cd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-27T14:18:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: fa70ec4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-27T15:06:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: fa70ec4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-27T17:33:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: fa70ec4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…ws#2713) Co-authored-by: Pranav Krishnan <[email protected]> Co-authored-by: Navin Soni <[email protected]>

change: expose num_clusters parameter for clarify shap in shapconfig

426c230

fix to pass lint check

0b03bc2

xgchena suggested changes Oct 18, 2021

View reviewed changes

src/sagemaker/clarify.py Show resolved Hide resolved

fixed shapconfig method signature to make backwards compatible

18c25b5

fixed shapconfig method signature to make backwards compatible

830e4bb

Merge branch 'master' into P50369404

8eb0e57

fix black check for unit tests

e87ad0b

fix black check again

b12677d

xgchena suggested changes Oct 21, 2021

View reviewed changes

src/sagemaker/clarify.py Show resolved Hide resolved

src/sagemaker/clarify.py Outdated Show resolved Hide resolved

src/sagemaker/clarify.py Show resolved Hide resolved

tests/unit/test_clarify.py Show resolved Hide resolved

src/sagemaker/clarify.py Outdated Show resolved Hide resolved

fixed shapconfig docstring, added shapconfig unit test

577a232

xgchena previously approved these changes Oct 21, 2021

View reviewed changes

pass lint-check

907f5b4

prkrishnan1 dismissed xgchena’s stale review via 907f5b4 October 21, 2021 23:55

navinsoni reviewed Oct 23, 2021

View reviewed changes

modifed error message for validating parameters

ee34924

prkrishnan1 dismissed xgchena’s stale review via ee34924 October 25, 2021 19:08

pass lint check

ed034ed

Merge branch 'master' into P50369404

c6c4c9d

navinsoni approved these changes Oct 26, 2021

View reviewed changes

shreyapandit approved these changes Oct 26, 2021

View reviewed changes

mufaddal-rohawala approved these changes Oct 26, 2021

View reviewed changes

Merge branch 'master' into P50369404

7ee3425

Merge branch 'master' into P50369404

435f8cd

Merge branch 'master' into P50369404

fa70ec4

navinsoni merged commit e5e0408 into aws:master Oct 27, 2021

change: expose num_clusters parameter for clarify shap in shapconfig #2713

change: expose num_clusters parameter for clarify shap in shapconfig #2713

Uh oh!

Conversation

prkrishnan1 commented Oct 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Oct 18, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 18, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 18, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 18, 2021

AWS CodeBuild CI Report

Uh oh!

Uh oh!

sagemaker-bot commented Oct 19, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 19, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 19, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 20, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 20, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 20, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 20, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 20, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 20, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 20, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 20, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Oct 20, 2021

AWS CodeBuild CI Report

Uh oh!

xgchena left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

Uh oh!

xgchena left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Oct 22, 2021

AWS CodeBuild CI Report

Uh oh!

navinsoni Oct 23, 2021

Choose a reason for hiding this comment

Uh oh!

navinsoni Oct 23, 2021

prkrishnan1 commented Oct 18, 2021 •

edited

Loading