Method utils.sts_regional_endpoint(region) always returns the global STS endpoint for all regions?

[GrabbenD]

Describe the bug
We'd like to return the regional STS endpoint with utils.sts_regional_endpoint(region), although this method always returns the global endpoint https://sts.amazonaws.com for all valid regions as well as non existent ones.

To reproduce
This can be easily reproduced with a one liner:

python3 -c 'import sagemaker; print(sagemaker.utils.sts_regional_endpoint("eu-west-1"))'

Expected behavior
We're expecting to get a regional endpoint such as: sts.eu-west-1.amazonaws.com if the method parameter is eu-west-1 instead of the global endpoint https://sts.amazonaws.com. The regional endpoints can be found in the AWS documentation: Managing AWS STS in an AWS Region.

System information

• SageMaker Python SDK version: 1.50.18.post0
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): N/A
• Framework version: N/A
• Python version: 3.7.5
• CPU or GPU: N/A
• Custom Docker image (Y/N): N

[knakad]

Thanks for reaching out!
Hmm.. a preliminary test on my end shows the correct behavior (https://sts.eu-west-1.amazonaws.com). I'll be diving a bit deeper into this momentarily, but while I do, I wonder if you have an old boto version. Can you run "pip freeze | grep boto" for me and shoot me what you see?

[GrabbenD]

@knakad Thanks for looking into this, here's the result:

$ pip freeze | grep boto
boto==2.49.0
boto3==1.12.15
botocore==1.15.15

Furthermore it seems like it's working correctly only if you use sudo?

$ sudo python3 -c 'import sagemaker; print(sagemaker.utils.sts_regional_endpoint("eu-west-1"))'
https://sts.eu-west-1.amazonaws.com

Any clue how we can we resolve this for regular users to avoid executing the code with sudo? (See comment below for solution)

[GrabbenD]

Solved!

From further testing it turns out the boto3 version was terribly outdated and caused this issue (due to a oversight on our part since pip3 =! pip when using python3):

$ pip3 freeze | grep boto
boto==2.49.0
boto3==1.9.188
botocore==1.12.247

Updating to the latest release solved the issue: pip3 install boto3 -U

Background
We're leaving a more detailed explanation for the orginal cause of this issue in case anyone else runs into this.

We tried to initialize and deploy a PyTorch model without declaring sagemaker_session in the object. This causes the SDK to use default_bucket(self) for the default Sagemaker session which sets self.boto_session.client() with endpoint_url=sts_regional_endpoint(region). When using outdated version of Boto3, the endpoint would get translated to https://sts.amazonaws.com which routes the request to us-east-1. Also, our default region for boto3 is set to eu-west-1 through AWS_REGION environment variable.. Because of this, we got the following error:

botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the GetCallerIdentity operation: Credential should be scoped to a valid region, not 'eu-west-1'

Seems like it's all good now, thanks for the help!