InvalidClientTokenId: The security token included in the request is invalid (govcloud)

[kuhe]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

(title) separating this from #5749 on behalf of @gilesvessey

SDK version number

@aws-sdk/___@3.503.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

n/a

Reproduction Steps

# .aws/config
[profile target]
region = us-gov-east-1
# .aws/credentials
[default]
source_profile = target
role_arn = [MASKED]
external_id = [MASKED]
region = us-gov-east-1
[target]
aws_access_key_id = [MASKED]
aws_secret_access_key = [MASKED]

import { STS } from "@aws-sdk/client-sts";

const client = new STS({});

console.log(await client.getCallerIdentity({}));

Observed Behavior

InvalidClientTokenId: The security token included in the request is invalid

Expected Behavior

200 response

Possible Solution

No response

Additional Information/Context

No response

[kuhe]

My latest setup on @aws-sdk/client-sts@3.504.0 and @aws-sdk/credential-provider-node@3.504.0:

# config
[default]
region = us-west-2
output = json
source_profile = default-target
role_arn = arn:aws:iam::...:role/Admin
external_id = actually-myself # I made this required

[profile default-target]
region = us-west-2

# credentials, long-lived from IAM user, no session token
[default-target]
aws_access_key_id=...
aws_secret_access_key=...

import { STS } from "@aws-sdk/client-sts";

const client = new STS({});

console.log(await client.getCallerIdentity({})); 
// this succeeds in v3.503.0, v3.503.1, and v3.504.0 of @aws-sdk/credential-provider-node, for me
// note: client-sts only has v3.501.0, v3.502.0, v3.504.0

I'm unable to reproduce the error so far, with this second revision being closer to your setup.

[gilesvessey]

Alright, I tested the above script using the exact same credential configuration, in a non-govcloud setting, and everything worked as expected with "@aws-sdk/client-sts": "3.504.0" and all other AWS imports removed from the package file. If I move back to govcloud, the issue unfortunately returns.

NON GOVCLOUD

gvessey@MacBook-Pro  ➤  unset "${!AWS@}"
✔  ~/src/giles-509/tools [509-health-assessment-utility|✚ 2… 12]
gvessey@MacBook-Pro  ➤  export AWS_SDK_LOAD_CONFIG=1
✔  ~/src/giles-509/tools [509-health-assessment-utility|✚ 2… 12]
gvessey@MacBook-Pro  ➤  aws sts get-caller-identity 
{
    "UserId": "[redacted]",
    "Account": "[redacted]",
    "Arn": "arn:aws:sts::[redacted]:assumed-role/[redacted]/botocore-session-1706812225"
}
✔  ~/src/giles-509/tools [509-health-assessment-utility|✚ 2… 12]
gvessey@MacBook-Pro  ➤  node test.js 
{
  '$metadata': {
    httpStatusCode: 200,
    requestId: 'cdca93ef-8b01-490b-b750-9b093de4d734',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  UserId: '[redacted]',
  Account: '[redacted]',
  Arn: 'arn:aws:sts::[redacted]:assumed-role/[redacted]/aws-sdk-js-1706812274855'
}

GOVCLOUD

✔  ~/src/giles-509/tools [509-health-assessment-utility|✚ 2… 12]
gvessey@MacBook-Pro  ➤  aws sts get-caller-identity
{
    "UserId": "[redacted]",
    "Account": "[redacted]",
    "Arn": "arn:aws-us-gov:sts::[redacted]:assumed-role/[redacted]/botocore-session-1706812323"
}
✔  ~/src/giles-509/tools [509-health-assessment-utility|✚ 2… 12]
gvessey@MacBook-Pro  ➤  node test.js 
/Users/gvessey/src/giles-509/tools/node_modules/@smithy/smithy-client/dist-cjs/index.js:838
  const response = new exceptionCtor({
                   ^

InvalidClientTokenId: The security token included in the request is invalid.
    at throwDefaultError (/Users/gvessey/src/giles-509/tools/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:20)
    at /Users/gvessey/src/giles-509/tools/node_modules/@smithy/smithy-client/dist-cjs/index.js:847:5
    at de_AssumeRoleCommandError (/Users/gvessey/src/giles-509/tools/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:375:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/gvessey/src/giles-509/tools/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /Users/gvessey/src/giles-509/tools/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /Users/gvessey/src/giles-509/tools/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/gvessey/src/giles-509/tools/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async Object.roleAssumer (/Users/gvessey/src/giles-509/tools/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:1516:43)
    at async /Users/gvessey/src/giles-509/tools/node_modules/@smithy/property-provider/dist-cjs/index.js:79:27 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: '747ff4fb-9d2e-4fea-ad2d-ae33cc256bac',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Type: 'Sender',
  Code: 'InvalidClientTokenId'
}

Node.js v18.18.2

[kuhe]

Can you try setting the region in two places like this?

import { STS } from "@aws-sdk/client-sts";
import { fromNodeProviderChain } from '@aws-sdk/credential-providers';

new STSClient({
  region: 'govcloud-region',
  credentials: fromNodeProviderChain({
    clientConfig: {
      region: 'govcloud-region',
    }
  }),
});

I explained outer/inner SDK clients in #5749 (comment) this comment.

To summarize, there are two STS clients in play, and when you don't use the global commercial partition in which STS uses us-east-1 by default, the region should be passed to both the inner and outer clients as shown above.

The reason there are two clients is that the "outer" client is the one you are initializing to make the getCallerIdentity request, and then there is an "inner" client that the SDK is initializing to fetch the credentials you need to make your request.

[kuhe]

I think it would be more user friendly if the inner client(s) used the same region as the user-initialized client(s) by default. I will see if we can implement that.

[gilesvessey]

Can you try setting the region in two places like this?

import { STS } from "@aws-sdk/client-sts";
import { fromNodeProviderChain } from '@aws-sdk/credential-providers';

new STSClient({
  region: 'govcloud-region',
  credentials: fromNodeProviderChain({
    clientConfig: {
      region: 'govcloud-region',
    }
  }),
});

I explained outer/inner SDK clients in #5749 (comment) this comment.

To summarize, there are two STS clients in play, and when you don't use the global commercial partition in which STS uses us-east-1 by default, the region should be passed to both the inner and outer clients as shown above.

The reason there are two clients is that the "outer" client is the one you are initializing to make the getCallerIdentity request, and then there is an "inner" client that the SDK is initializing to fetch the credentials you need to make your request.

Yup, this worked.