Unable to cast object of type 'generatedProxy_2' to type 'Microsoft.PowerPlatform.Dataverse.Client.IOrganizationServiceAsync'

[sdhz236]

I‘m getting this issue occasionally for azure function app, I'm using the latest verion of Microsoft.PowerPlatform.Dataverse.Client（0.6.1）.

Full Exception :

Microsoft.PowerPlatform.Dataverse.Client.Utils.DataverseConnectionException : Failed to connect to Dataverse ---> System.AggregateException : One or more errors occurred. (Unable to cast object of type 'generatedProxy_2' to type 'Microsoft.PowerPlatform.Dataverse.Client.IOrganizationServiceAsync'.) ---> Unable to cast object of type 'generatedProxy_2' to type 'Microsoft.PowerPlatform.Dataverse.Client.IOrganizationServiceAsync'.

at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)

at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)

at System.Threading.Tasks.Task`1.get_Result()

at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.GetCachedService(ConnectionService& ConnectionObject)

at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.IntilizeService(ConnectionService& ConnectionObject)

at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection(Object externalOrgServiceProxy,AuthenticationType requestedAuthType,String hostName,String port,String orgName,NetworkCredential credential,String userId,SecureString password,String domain,String Geo,String claimsHomeRealm,Boolean useSsl,Boolean useUniqueInstance,OrganizationDetail orgDetail,String clientId,Uri redirectUri,PromptBehavior promptBehavior,OrganizationWebProxyClientAsync externalOrgWebProxyClient,String certificateThumbPrint,StoreName certificateStoreName,X509Certificate2 certificate,Uri instanceUrl,Boolean isCloned,Boolean useDefaultCreds,Version incomingOrgVersion,ILogger externalLogger,String tokenCacheStorePath)

---> (Inner Exception #0) System.InvalidCastException : Unable to cast object of type 'generatedProxy_2' to type 'Microsoft.PowerPlatform.Dataverse.Client.IOrganizationServiceAsync'.

at async Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.InitServiceAsync()<---

End of inner exception

at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection(Object externalOrgServiceProxy,AuthenticationType requestedAuthType,String hostName,String port,String orgName,NetworkCredential credential,String userId,SecureString password,String domain,String Geo,String claimsHomeRealm,Boolean useSsl,Boolean useUniqueInstance,OrganizationDetail orgDetail,String clientId,Uri redirectUri,PromptBehavior promptBehavior,OrganizationWebProxyClientAsync externalOrgWebProxyClient,String certificateThumbPrint,StoreName certificateStoreName,X509Certificate2 certificate,Uri instanceUrl,Boolean isCloned,Boolean useDefaultCreds,Version incomingOrgVersion,ILogger externalLogger,String tokenCacheStorePath)

at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.ConnectToService(String connectionString,ILogger logger)

at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient..ctor(String dataverseConnectionString,ILogger logger)

at myFunction.Startup.<>c.b__2_0(IServiceProvider s) at D:\XXX\AzureFunction\AzureFunctions\myFunction\Startup.cs : 16

at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.b__0(IResolverContext r) at D:\a_work\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs : 156

at DryIoc.Registrator.<>c__DisplayClass27_0.b__0(IResolverContext r) at D:\a_work\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 4550

at System.Linq.Expressions.Interpreter.FuncCallInstruction`3.Run(InterpretedFrame frame)

at System.Linq.Expressions.Interpreter.Interpreter.Run(InterpretedFrame frame)

at System.Linq.Expressions.Interpreter.LightLambda.Run(Object[] arguments)

at Thunk(Func`2 ,IResolverContext )

at DryIoc.Factory.<>c__DisplayClass26_0.b__2() at D:\a_work\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6605

at DryIoc.Scope.TryGetOrAdd(ImMap`1 items,Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at D:\a_work\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7849

at DryIoc.Scope.GetOrAdd(Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at D:\a_work\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7834

at DryIoc.Factory.ApplyReuse(Expression serviceExpr,Request request) at D:\a_work\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6604

at DryIoc.Factory.GetExpressionOrDefault(Request request) at D:\a_work\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6564

at DryIoc.Factory.GetDelegateOrDefault(Request request) at D:\a_work\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6634

at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) at D:\a_work\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7740

at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType,IfUnresolved ifUnresolved) at D:\a_work\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 211

at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType,IfUnresolved ifUnresolved) at D:\a_work\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 196

at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedServiceProvider.GetService(Type serviceType) at D:\a_work\1\s\src\WebJobs.Script.WebHost\DependencyInjection\ScopedServiceProvider.cs : 25

at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp,Type type,Type requiredBy,Boolean isDefaultParameterRequired)

at lambda_method(Closure ,IServiceProvider ,Object[] )

at Microsoft.Azure.WebJobs.Host.Executors.DefaultJobActivator.CreateInstance[T](IServiceProvider serviceProvider) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\DefaultJobActivator.cs : 37

at Microsoft.Azure.WebJobs.Host.Executors.DefaultJobActivator.CreateInstance[T](IFunctionInstanceEx functionInstance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\DefaultJobActivator.cs : 32

at Microsoft.Azure.WebJobs.Host.Executors.ActivatorInstanceFactory`1.<>c__DisplayClass1_1.<.ctor>b__0(IFunctionInstanceEx i) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\ActivatorInstanceFactory.cs : 20

at Microsoft.Azure.WebJobs.Host.Executors.ActivatorInstanceFactory`1.Create(IFunctionInstanceEx functionInstance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\ActivatorInstanceFactory.cs : 26

at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.CreateInstance(IFunctionInstanceEx functionInstance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs : 44

at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ParameterHelper.Initialize() at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 799

at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 104

[MoranShusheng]

We have encountered the same problem.
There are a lot of errors.
Do you have any progress on this problem?

[MattB-msft]

We have not been able to reliably duplicate the problem at this point and are speculating on what is going on.
it would be extremely helpful if you could provide us a repro of this issue or more information about what is going on when it happens?

thanks
MattB

[MortenRoemer]

We have the same issue here. Sadly the project I'm working on is proprietary but I can reproduce the error in a reliable way:

Framework: .NET 6
Function Runtime Version: latest

Steps:

1. Create a function app that has a startup function
2. Initialize a ServiceClient with Client/Secret authentication within that startup function

This produces the mentioned error for us every time we start up the function app.
Any pointers what the underlying issues could be? With the right pointers I could try to fix the issue myself and create a pull-request

Edit: We are working with the european tenants of Dynamics so our instances have the pattern <instancename>.crm4.dynamics.com

[MattB-msft]

The client does initialize? Or does it fail and you have a non working client?

[MortenRoemer]

The Client fails to initialize. After that every call to Public members results in NullReferenceExceptions

[MattB-msft]

Can you paste your setup block here?
I assume you have a startup.cs with a public override void Configure(IFunctionsHostBuilder builder) your adding the setup call too?

[MattB-msft]

As an example,... this is what one of our function test cases' startup.cs's looks like:
This is Functions 4.0. ( this is not a formal example, but it is from an internal function we are using )

[assembly: FunctionsStartup(typeof(ExampleFunction_DI.Startup))]
namespace ExampleFunction_DI
{
    internal class Startup : FunctionsStartup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {
            builder.Services.AddLogging();
            builder.Services.AddSingleton<ServiceClient>(sp =>
            {
                var logFactory = sp.GetService<ILoggerFactory>();
                var logger = logFactory.CreateLogger(typeof(ServiceClient));
                return GetClient(logger);
            });

        }

        static ServiceClient GetClient(ILogger logger = null)
        {
            logger?.LogInformation("Connecting to Dataverse");
            var connectionString = $"{Environment.GetEnvironmentVariable("ConnectionStringBase")};AppId={Environment.GetEnvironmentVariable("DataverseAppId")};ClientSecret={Environment.GetEnvironmentVariable("DataverseAppIdSecret")};Url={Environment.GetEnvironmentVariable("InstanceUri")}";
            ServiceClient client = null;
            client = new ServiceClient(connectionString, logger);
            if (client.IsReady)
            {
                logger?.LogInformation("Connected to Dataverse...");
                return client;
            }
            else
                throw client.LastException;
        }

    }
}

[MortenRoemer]

Our startup function is a little bit simpler:

public class Startup : FunctionsStartup
{
    public override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
    {
        builder.ConfigurationBuilder
            .AddUserSecrets(Assembly.GetExecutingAssembly(), true);

        base.ConfigureAppConfiguration(builder);
    }
    
    public override void Configure(IFunctionsHostBuilder builder)
    {
        LoadServices(builder.Services);
    }

    private static void LoadServices(IServiceCollection services)
    {
        services
            .AddSingleton<ServiceClient>(ConfigureDataverseClient);
    }

    private static ServiceClient ConfigureDataverseClient(IServiceProvider serviceProvider)
    {
        var client = new ServiceClient(new Uri(Config.DataverseUrl.Value!), Config.DataverseClientId.Value!, Config.DataverseClientSecret.Value!, true);
        return client;
    }
}

[MortenRoemer]

EDIT: This workaround reduced the likelyhood of this issue but not prevented it. See post from 2022-05-09 for more info

But we found a workaround: When we handle the client/secret authentication ourselves it works without issues:

Here is our workaround startup function:

public class Startup : FunctionsStartup
{
    public override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
    {
        builder.ConfigurationBuilder
            .AddUserSecrets(Assembly.GetExecutingAssembly(), true);

        base.ConfigureAppConfiguration(builder);
    }
    
    public override void Configure(IFunctionsHostBuilder builder)
    {
        LoadServices(builder.Services);
    }

    private static void LoadServices(IServiceCollection services)
    {
        services
            .AddSingleton<ServiceClient>(ConfigureDataverseClient);
    }

    private static ServiceClient ConfigureDataverseClient(IServiceProvider serviceProvider)
    {
        var httpClient = new HttpClient();
        var authHandler = new DataverseAuthHandler(
            httpClient, 
            Config.DataverseTenantId.Value!,
            Config.DataverseUrl.Value!,
            Config.DataverseClientId.Value!, 
            Config.DataverseClientSecret.Value!
        );
        var client = new ServiceClient(new Uri(Config.DataverseUrl.Value!), _ => authHandler.RetrieveToken(_));
        return client;
    }
}

And our helper class that handles the client/secret authentication:

public class DataverseAuthHandler
{
    public DataverseAuthHandler(HttpClient httpClient, string tenantId, string instanceUrl, string clientId, string clientSecret)
    {
        HttpClient = httpClient;
        LoginUrl = $"https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token";
        InstanceUrl = instanceUrl;
        ClientId = clientId;
        ClientSecret = clientSecret;
    }
    
    private HttpClient HttpClient { get; }
    
    private string LoginUrl { get; }
    
    private string InstanceUrl { get; }
    
    private string ClientId { get; }
    
    private string ClientSecret { get; }
    
    private string? Token { get; set; }
    
    private DateTime? TokenValidUntil { get; set; }
    
    private bool IsTokenStillValid 
        => Token is not null && DateTime.UtcNow < TokenValidUntil!.Value;

    public async Task<string> RetrieveToken(string _)
    {
        if (IsTokenStillValid)
            return Token!;

        var tokenResult = await LogIn(InstanceUrl);
        Token = tokenResult.AccessToken;
        TokenValidUntil = DateTime.UtcNow + TimeSpan.FromMinutes(25);
        return Token;
    }

    private async Task<TokenResult> LogIn(string instanceUrl)
    {
        var scope = instanceUrl + ".default";
        var message = new HttpRequestMessage(HttpMethod.Post, LoginUrl)
        {
            Content = new FormUrlEncodedContent(
                new Dictionary<string, string>
                {
                    { "grant_type", "client_credentials" },
                    { "client_id", ClientId },
                    { "client_secret", ClientSecret },
                    { "scope", scope }
                }
            )
        };

        var response = await HttpClient.SendAsync(message);
        response.EnsureSuccessStatusCode();
        return JsonSerializer.Deserialize<TokenResult>(await response.Content.ReadAsStreamAsync())!;
    }

    private class TokenResult
    {
        [JsonPropertyName("token_type")]
        public string TokenType { get; set; } = null!;

        [JsonPropertyName("expires_in")]
        public int ExpiresIn { get; set; }

        [JsonPropertyName("ext_expires_in")]
        public int ExtExpiresIn { get; set; }

        [JsonPropertyName("access_token")]
        public string AccessToken { get; set; } = null!;
    }
}

[MattB-msft]

Are you using the 0.6.6 client? If not can you retest on the please .

[MortenRoemer]

I've tested it initially on the 0.6.6 release. There I encountered the error

[MattB-msft]

Thanks, can you tell us about you hosting setup,
What version of azure functions? What version of the azure functions library (nuget) and dependency injection your using?
What OS host?
Dedicated or shared sub for azure?

Thanks

[MortenRoemer]

Here:

• Azure Function Runtime: latest
• Microsoft.NET.Sdk.Functions: 4.1.0
• Microsoft.Extensions.Configuration 6.0.1
• Host OS: Linux
• dedicated subscription

Hope that helps

[MattB-msft]

We are, sofar, unable to duplicate this issue.

Can you add an ILogger to your function and capture logs for us of the failure?
To do that... Add the Ilogger ( Created As I showed above, the constructor your using will accept an ILogger as the final parameter ) and include in your host.json

...
  "logging": {
    "logLevel": {
      "Microsoft.PowerPlatform.Dataverse.Client.ServiceClient": "Trace"
    },
...

if you are not comfortable sharing the output here on Git,
please go here: Nuget: Microsoft.PowerPlatform.Dataverse.Client
and hit contact owner - middle of the page on the right and send an email referring to this issue

I will respond directly to you via email and you can reply with the log

Thanks for your help in helping us track this down.

[TomWinton]

This may or may not help but wanted to confirm that I was also getting this issue, was using also using version 6.6, after seeing this and not wanting to try MortenRoemers clever workaround I tried going back down to 5.17 with same result.

Azure Function Runtime: latest
Microsoft.NET.Sdk.Functions: 4.1.0
Host OS: Windows
Consumption Plan
.Net 6

Initially I was generating a new ServiceClient each time a request was required with a connection string.

Restarting the function would temporarily resolve the issue, the issue would then occur again after a period of time and then continue to happen, given it occurred in line with a spike in requests to the function I made an assumption (which i think is correct) that it occurs when the function auto scales out.

I implemented an easier but way stupiderer workaround which was that I have a declared a static ServiceClient and during startup I create the assign it.

Now, when I stare at the live logs on Azure I can see that this issue still occurs, but because it occurs during startup Microsoft helpfully abort this scale out attempt and makes a new instance (or maybe just abandons the scale out attempt entirely and restarts the service) which ends up working so when my functions needs it it doesn't bork out.

Either way, seeing as now I make the whole thing crap out when it tries to scale out the issue has to be something to do with reusing invalid info from another instance.