[AI Evaluation] MEAI.Evaluation.Reporting includes provider URI in cache key, breaking usage in team scenarios

[SteveSandersonMS]

Description

Is there an important reason for ReportingConfiguration.cs's GetCachingKeysForChatClient to explicitly include the ChatClientMetadata.ProviderUri value in the cache key?

The scenario I'm concerned with is:

• A team wants to store the cache in source control so that each developer doesn't have to invoke and wait for a huge number of LLM calls when running evaluation or integration tests. On each run, they will only have to wait for the LLM calls whose inputs they have modified in their code change.
• Likewise in CI, the team doesn't want to set AI service credentials at all, but will rely on all the LLM calls being resolved from the source controlled cache. They will (correctly) get test failures if anyone fails to update the cache with their PRs, which is good.
• However, each developer on the team has their own separate Azure OpenAI deployments and hence must each configure a different endpoint. And the CI server doesn't have any endpoint to configure.

Including the endpoint URL in the cache key means you'll never get any cache hits in this scenario, and it isn't possible to use this technique to verify integration tests in CI.

Proposal: don't include the ProviderUri in the cache key, or at least provide an easy way to opt out from this.

If I'm misunderstanding the intended use case for the cache, please let me know. But I think being able to store it in source control is likely the most valuable usage pattern.

Reproduction Steps

var reportingConfig = DiskBasedReportingConfiguration.Create(
    "some/path", [], new ChatConfiguration(myChatClient));
var scenarioRun = await reportingConfig.CreateScenarioRunAsync("SomeName");
var chatClient = scenarioRun.ChatConfiguration!.ChatClient;
var result = await chatClient.GetResponseAsync("What's 1+1?");

Expected behavior

Cache entry should not vary by myChatClient.GetService<ChatClientMetadata>().ProviderUri

Actual behavior

... but it does.

Regression?

No response

Known Workarounds

Technically you could create a custom IChatClient wrapper that hides the ChatClientMetadata from MEAI.Evaluations but that is harmful in other ways.

Configuration

No response

Other information

No response

[shyamnamboodiripad]

@SteveSandersonMS The original reason to include provider URI was to ensure that if someone switches to a different endpoint, then that should be considered a significant enough change and warrant discarding the existing cached entries (especially in CI).

However, the scenario you raise above is interesting and does make sense. I am happy to go ahead and remove the provider URI from our default cache key implementation. This would leave the providerName and modelId both of which seem ok to continue to include. (cc @peterwald)

We also already have a mechanism that consumers can use to supply additional cachingKeys (as part of the ReportingConfiguration). So, a consumer could still include the provider URI that way if they wanted to. Thanks for the feedback!

[peterwald]

@SteveSandersonMS @shyamnamboodiripad I hadn't thought about this scenario before, either. I think it's a good enough reason to remove the URI from the cache key. We were thinking originally that there could be settings that are different between two endpoints that we might want to honor in the cache, but I agree that since the user can add items to the key themselves there is an easy workaround.