Skip to content

[BUG] Remote connector incorrectly SigV4 signs requests to certain remote services #4100

@q-andy

Description

@q-andy

What is the bug?
I've noticed that some remote connector requests that are signed with SigV4 fail with 403s due to signature mismatch. After troubleshooting for a bit, I noticed that we're using Aws4Signer to sign requests, which is deprecated in AWS SDKv2. We're already using AWS SDKv2, so we should be using AwsV4HttpSigner instead.

Docs:
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/signer/Aws4Signer.html
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/auth/aws/signer/AwsV4HttpSigner.html

Replacing the signer on my local fixes the bad signature error, despite not changing anything else (credentials, headers, connectors are the same)

How can one reproduce the bug?
Steps to reproduce the behavior:

When calling AWS OpenSearch Serverless USING local ml commons/opensearch (the remote connector is NOT on serverless, the connector is on open source OpenSearch), it gives 403s when using Aws4Signer, but not with AwsV4HttpSigner, even with correct credentials. To verify

  1. Create a serverless collection
  2. Set up a resource to call from local opensearch cluster (e.g. sparse embedding model)
  3. Get credentials
  4. Spin up local opensearch cluster and create a connector:
POST /_plugins/_ml/connectors/_create
{
    "name": "",
    "description": "",
    "version": 1,
    "protocol": "aws_sigv4",
    "parameters": {
        "service_name": "aoss",
        "region": "us-east-1"
    },
    "credential": {
        "access_key": "credential",
        "secret_key": "credential",
        "session_token": "credential"
    },
    "actions": [
        {
            "action_type": "predict",
            "method": "POST",
            "headers": {
                "content-type": "application/json"
            },
            "url": "https://<collection>.us-east-1.aoss.amazonaws.com/_plugins/_ml/models/<model>/_predict?algorithm=text_embedding",
            "request_body": "{\"text_docs\": ${parameters.input}}",
            "post_process_function": "connector.post_process.mlcommons.passthrough"
        }
    ]
}

Then register and predict the model.

What is the expected behavior?
Correct credentials should produce correct signature, but you get a 403. Using AwsV4HttpSigner fixes it.

What is your host/environment?

  • OS: Mac
  • Version 3.2
  • Plugins

Do you have any screenshots?
n/a

Do you have any additional context?
n/a

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions