Skip to content

Conversation

dbschmigelski
Copy link
Member

Description

Currently only Bedrock is being exercised in our integration tests. However, the Strands team has secured API keys for various providers. These keys are stored in Secrets Manager are loaded before all tests. The Secrets Manager Secret name is hidden using GitHub repository secrets. However, the expected form is

{
   {model_provider_1}: {key1},
    {model_provider_2}: {key2},
}

Tests are skipped when the environment variable is missing. To prevent regressions there is now an added validation where if any of the following are not set as env vars AND we are executing within a GitHub Actions workflow then we fail

    required_providers = {
        "ANTHROPIC_API_KEY",
        "COHERE_API_KEY",
        "MISTRAL_API_KEY",
        "OPENAI_API_KEY",
        "WRITER_API_KEY",
    }

The test in conformance.py was not actually being executed because the file did not have a test_ prefix. To address this it was renamed tests_integ/models/test_conformance.py. Additionally, because it was not being executed a bug was left in remaining from the pre 1.0.0 release where the Model ABC was moved from types/ to models/.

Related Issues

#237

Type of Change

Other (please describe): Enable testing of all providers

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli

Tests were run in a testing repository to verify the workflows succeed

  • I ran hatch run prepare

Checklist

  • I have read the CONTRIBUTING document
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

zastrowm
zastrowm previously approved these changes Jul 21, 2025
@dbschmigelski
Copy link
Member Author

Noting that Secure Integration test / check-access-and-checkout (pull_request_target) is failing because we expect STRANDS_TEST_API_KEYS_SECRET_NAME to be set in the workflow. But, because the workflow is pull_request_target it is using the existing one which does not yet set STRANDS_TEST_API_KEYS_SECRET_NAME.

Will need to get a separate PR out just to expose it

@dbschmigelski
Copy link
Member Author

The linting change is failing because of a breaking change made by A2A which switched naming form camel to snake case https://github.com/a2aproject/a2a-python/releases/tag/v0.2.16

@zastrowm zastrowm added the area-provider Related to model providers label Jul 22, 2025
@dbschmigelski dbschmigelski merged commit 022ec55 into strands-agents:main Jul 22, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-provider Related to model providers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants