Bug: change: use regional endpoint when creating AWS STS client #1026

[adrian-chang]

Please fill out the form below.

System Information

• Python 3.6.9
• Sagemaker SDK 1.39.0

Describe the problem

PR #1026 introduced a bug by not using a scheme for a particular STS endpoint

Minimal repro / logs

sagemaker.get_execution_role(sagemaker_session)

Leads to this stack trace

  File "/Library/Caches/virtualenvs/gocentral-ml-sagemaker-0XKWRhha-py3.6/lib/python3.6/site-packages/sagemaker/session.py", line 1386, in get_caller_identity_arn
    "sts", endpoint_url=sts_regional_endpoint(self.boto_region_name)
  File "/Library/Caches/virtualenvs/gocentral-ml-sagemaker-0XKWRhha-py3.6/lib/python3.6/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/Library/Caches/pypoetry/virtualenvs/gocentral-ml-sagemaker-0XKWRhha-py3.6/lib/python3.6/site-packages/botocore/session.py", line 839, in create_client
    client_config=config, api_version=api_version)
  File "/Library/Caches/pypoetry/virtualenvs/gocentral-ml-sagemaker-0XKWRhha-py3.6/lib/python3.6/site-packages/botocore/client.py", line 86, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/Library/Caches/pypoetry/virtualenvs/gocentral-ml-sagemaker-0XKWRhha-py3.6/lib/python3.6/site-packages/botocore/client.py", line 328, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/Library/Caches/pypoetry/virtualenvs/gocentral-ml-sagemaker-0XKWRhha-py3.6/lib/python3.6/site-packages/botocore/args.py", line 85, in get_client_args
    client_cert=new_config.client_cert)
  File "/Library/Caches/pypoetry/virtualenvs/gocentral-ml-sagemaker-0XKWRhha-py3.6/lib/python3.6/site-packages/botocore/endpoint.py", line 261, in create_endpoint
    raise ValueError("Invalid endpoint: %s" % endpoint_url)
ValueError: Invalid endpoint: sts.us-west-2.amazonaws.com

The endpoint.py requires a scheme and non is provided via the sts_regional_endpoint method.

[adrian-chang]

More details: https://github.com/boto/botocore/blob/develop/botocore/utils.py#L832-L853 from botocore prevents this change from working

[laurenyu]

Thanks for the detailed bug report, and apologies for the trouble this has caused. I've submitted a PR to fix this: #1035

[diegarba14]

Hey the fix of this bug was already merged and deployed?

[laurenyu]

not deployed yet - will do so tomorrow morning. In the meantime, please downgrade your SDK version to 1.38.6. Sorry for the inconvenience!

[diegarba14]

I am trying to use jupyter notebook and is throwing same error, this issue will be fixed when you deployed the fixes?

[dnorth98]

How does one downgrade the SDK in the jupyter notebook? Just spent a chunk of time on this issue trying to get the demo working.

[diegarba14]

Yeah, I am block now for the same thing, cannot use jupyter notebook even for demo notebooks, this issue is a headache, totally crazy a company as Amazon has this type of bugs

[ivenzor]

The default sagemaker package still has de issue (1.39).

But it seems it's working if you downgrade to 1.38.6:
pip install sagemaker==1.38.6

[bscholesboogie]

I'm getting the same error even after downgrading to 1.38.6.

[diegarba14]

Any Estimated time for this issue?

[ivenzor]

It's working for me:

1. open the terminal and activate the right conda env used in jupyter
2. pip install sagemaker==1.38.6
3. reload the kernel in jupyter and import sagemaker package
4. confirm that the 1.38 version is loaded
5. the get_execution_role() should be working now.

[bscholesboogie]

Apparently the second time is the charm. Downgraded to 1.38.6 again and now it works.