Exception : Binary format of the specified custom attribute was invalid when reading the configuration

[NicolasREY69330]

Description
Hi,
when upgrading to these packages

    <PackageReference Include="Serilog" Version="2.12.0" />
    <PackageReference Include="Serilog.Extensions.Hosting" Version="5.0.1" />
    <PackageReference Include="Serilog.Extensions.Logging" Version="3.1.0" />
    <PackageReference Include="Serilog.Settings.Configuration" Version="3.4.0" />
    <PackageReference Include="Serilog.Sinks.Console" Version="4.1.0" />
    <PackageReference Include="Serilog.Sinks.ElasticSearch" Version="8.4.1" />
    <PackageReference Include="Serilog.Sinks.Seq" Version="5.2.1" />
    <PackageReference Include="StackExchange.Redis" Version="2.6.66" />

I get an exception when creating the logger with

Log.Logger = LoggerConfiguration()
            .ReadFrom.Configuration(config)
            .Enrich.WithElasticApmCorrelationInfo()
            .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://xxxxxxxxxxxxxxxxxxx"))
            {
                AutoRegisterTemplate = true,
                IndexFormat = "mslogs-{0:yyyy.MM.dd}",
                DetectElasticsearchVersion = true,
                RegisterTemplateFailure = RegisterTemplateRecovery.IndexAnyway,
                AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
                FailureCallback = e => Console.WriteLine($"Unable to submit event {e?.RenderMessage()} to ElasticSearch. Exception : " + e?.Exception?.ToString()),
                EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
                                        EmitEventFailureHandling.WriteToFailureSink |
                                        EmitEventFailureHandling.RaiseCallback,
                BufferCleanPayload = (failingEvent, statuscode, exception) =>
                {
                    dynamic e = JObject.Parse(failingEvent);
                    return JsonConvert.SerializeObject(new Dictionary<string, object>()
                        {
                            { "action", "DeniedByElasticSearch"},
                            { "@timestamp",e["@timestamp"]},
                            { "level","Error"},
                            { "message","Error: "+e.message},
                            { "messageTemplate",e.messageTemplate},
                            { "failingStatusCode", statuscode},
                            { "failingException", exception}
                        });
                },
                CustomFormatter = new EcsTextFormatter()
            })
            .CreateLogger();

with the assumed configuration:

  "Serilog": {
    "Using": [],
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "System": "Information",
        "Elastic": "Warning",
        "Apm": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Console"
      }
    ],
    "Enrich": [
      "FromLogContext",
      "WithMachineName",
      "WithProcessId",
      "WithThreadId"
    ],
    "Properties": {
      "ApplicationName": "myapplication"
    }
  }

It throws the following exception :

Unhandled exception. System.Reflection.CustomAttributeFormatException: Binary format of the specified custom attribute was 
invalid. 
at System.Reflection.CustomAttributeEncodedArgument.ParseAttributeArguments(IntPtr pCa, Int32 cCa, 
CustomAttributeCtorParameter[]& CustomAttributeCtorParameters, CustomAttributeNamedParameter[]& 
CustomAttributeTypedArgument, RuntimeAssembly assembly) 
at System.Reflection.RuntimeCustomAttributeData..ctor(RuntimeModule scope, MetadataToken caCtorToken, ConstArray& blob)  
at System.Reflection.RuntimeCustomAttributeData.GetCustomAttributes(RuntimeModule module, Int32 tkTarget) 
at System.Reflection.RuntimeCustomAttributeData.GetCustomAttributesInternal(RuntimeMethodInfo target) 
at System.Reflection.RuntimeMethodInfo.GetCustomAttributesData()   at System.Reflection.MemberInfo.get_CustomAttributes()  
at Serilog.Settings.Configuration.ConfigurationReader. FindConfigurationExtensionMethods>g__HasExtensionAttribute|32_0(MethodInfo m) 
at Serilog.Settings.Configuration.ConfigurationReader.<>c.<FindConfigurationExtensionMethods>b__32_3(MethodInfo m) 
at System.Linq.Utilities.<>c__DisplayClass1_0`1.<CombinePredicates>b__0(TSource x)
at System.Linq.Enumerable.WhereEnumerableIterator`1.ToList() 
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) 
at Serilog.Settings.Configuration.ConfigurationReader.FindConfigurationExtensionMethods(IReadOnlyCollection`1 
configurationAssemblies, Type configType)
at Serilog.Settings.Configuration.ConfigurationReader.FindEventEnricherConfigurationMethods(IReadOnlyCollection`1 
configurationAssemblies) 
at Serilog.Settings.Configuration.ConfigurationReader.ApplyEnrichment(LoggerConfiguration 
loggerConfiguration)
at Serilog.Settings.Configuration.ConfigurationReader.Configure(LoggerConfiguration 
loggerConfiguration)
at Serilog.Configuration.LoggerSettingsConfiguration.Settings(ILoggerSettings settings)
at Serilog.ConfigurationLoggerConfigurationExtensions.Configuration(LoggerSettingsConfiguration settingConfiguration, 
IConfiguration configuration, String sectionName, DependencyContext dependencyContext)
at Serilog.ConfigurationLoggerConfigurationExtensions.Configuration(LoggerSettingsConfiguration settingConfiguration, 
IConfiguration configuration, DependencyContext dependencyContext)
at ITF.SharedLibraries.Logging.LoggerFactory.GetELKLogger(IConfiguration config, String varEnv)
at MyApplication.Startup.ConfigureServices(IServiceCollection services) in /src/MyApplication/src/Startup.cs:line 46 
at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.AspNetCore.Hosting.MethodInfoExtensions.InvokeWithoutWrappingExceptions(MethodInfo methodInfo, Object obj, Object[] parameters)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services) 
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0. <Invoke>g__Startup|0(IServiceCollection serviceCollection) 
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services) 
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, 
IServiceCollection services, Object instance) 
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.
<UseStartup>b__0(HostBuilderContext context, IServiceCollection services) 
at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
at Microsoft.Extensions.Hosting.HostBuilder.Build() 
at MyApplication.Program.Main(String[] args) in /src/MyApplication/src/Program.cs:line 12

Downgrading to previous versions solved the problem

    <PackageReference Include="Serilog" Version="2.11.0" />
    <PackageReference Include="Serilog.Extensions.Hosting" Version="5.0.1" />
    <PackageReference Include="Serilog.Extensions.Logging" Version="3.1.0" />
    <PackageReference Include="Serilog.Settings.Configuration" Version="3.3.0" />
    <PackageReference Include="Serilog.Sinks.Console" Version="4.0.1" />
    <PackageReference Include="Serilog.Sinks.ElasticSearch" Version="8.4.1" />
    <PackageReference Include="Serilog.Sinks.Seq" Version="5.1.1" />
    <PackageReference Include="StackExchange.Redis" Version="2.6.48" />

Reproduction
The problem occurred only in our Production environment, it was ok in local and in uat environment.
We connect against a containerized Elasticsearch (v 7.17.1)

Expected behavior
No crash on logger creation

Additional context
Kubernetes hosting in AKS

[bartelink]

While it's possible this is a bug in Serilog as you're speculatively suggesting, the greater likelihood is that you have mismatched or otherwise mangled DLLs in your prod scenario.

Also, this should be obvious but... can you please format the exception stacktrace in the OP properly - nobody is going to read it if you leave your homework messy like that!

Resolving the issue is going to be the normal process of narrowing things down and ruling things out. At the point where you have a failing test that's not "here are 57 interlinked things that together give me a random problem", it becomes a workable issue for this repo. Until then, StackOverflow is going to be the best help (Most people watching this repo also watch the SO serilog tag, which is why the issue template suggests putting stuff there first).

[NicolasREY69330]

@bartelink I agree that the exception format is not readable, I'm going to fix this.

For the rest, well I have two microservices that had the same systematic behavior when upgrading the Serilog* nuget and which "recovered" after downgrading them. My intention is not to put blame on Serilog package which I use for years in production now, but really to understand what happened, maybe it might be a bug or a combination of circumstance (between Elk version/data/settings + Serilog) I don't know, and that's why I require the help of the community. Sorry if I choose the wrong GH issue template or channel for this, I'll try with SO also.

[bartelink]

For the rest, well I have two microservices that had the same systematic behavior when upgrading the Serilog* nuget and which "recovered" after downgrading them

Best to close this and move it to SO then; feel free to post a link to the SO post as you close - clearly your build+deploy pipeline is consistently leading to a mismatched or mangled set of DLLs. That can be either a matter of turning up warning levels re mismatched component versions, and/or figuring out what's manglign and/or mismatching them after the fact. Wild guess is that some part of your system is still using older versions and they're all getting merged, with the outcome that new stuff is touching old stuff.

Given the prevalence of Serilog across all sorts of companies and environments, an actual minimal repro would show up in this repro pretty soon if there was an actual issue (and that'd be turned into a failing regression test first before being fixed)

[NicolasREY69330]

Thanks for your help, here is the SO link https://stackoverflow.com/questions/73987270/serilog-to-elasticsearch-exception-on-creating-the-builder

[skomis-mm]

Hi @NicolasREY69330 ,
does downgrading only Serilog.Settings.Configuration to 3.3.0 resolves the issue?
If yes, please re-open, looks like a bug here

[skomis-mm]

StackTrace has link to this change. Not sure what happens here, need to investigate

[NicolasREY69330]

Hi @NicolasREY69330 , does downgrading only Serilog.Settings.Configuration to 3.3.0 resolves the issue? If yes, please re-open, looks like a bug here

I'll try on next week and keep you informed thanks for your help

[NicolasREY69330]

@skomis-mm
I just retried, with the following packages, it throws the exception

<PackageReference Include="Serilog" Version="2.12.0" />
<PackageReference Include="Serilog.Extensions.Hosting" Version="5.0.1" />
<PackageReference Include="Serilog.Extensions.Logging" Version="3.1.0" />
<PackageReference Include="Serilog.Settings.Configuration" Version="3.4.0" />
<PackageReference Include="Serilog.Sinks.Console" Version="4.1.0" />
<PackageReference Include="Serilog.Sinks.ElasticSearch" Version="8.4.1" />
<PackageReference Include="Serilog.Sinks.Seq" Version="5.2.1" />

and when downgrading the package Serilog.Settings.Configuration from 3.4.0 to 3.3.0 it works as expected

<PackageReference Include="Serilog" Version="2.12.0" />
<PackageReference Include="Serilog.Extensions.Hosting" Version="5.0.1" />
<PackageReference Include="Serilog.Extensions.Logging" Version="3.1.0" />
<PackageReference Include="Serilog.Settings.Configuration" Version="3.3.0" />
<PackageReference Include="Serilog.Sinks.Console" Version="4.1.0" />
<PackageReference Include="Serilog.Sinks.ElasticSearch" Version="8.4.1" />
<PackageReference Include="Serilog.Sinks.Seq" Version="5.2.1" />

[nunorelvao]

Hi all, just to confirm this issue happened to me exactly the same:

• locally direct debug no problem.
• locally in docker linux also no problem.
• as soon as it was published on the cloud Kubernetes it fails with this error.

The fix for now is to downgrade from 3.4.0 to:
<PackageReference Include="Serilog.Settings.Configuration" Version="3.3.0" />

[nblumhardt]

👍

My bet is that you're building for one version of .NET but deploying to a slightly earlier one. How are your apps being published (command lines etc.) and what's the exact dotnet version hosting them at runtime?

Another possibility is that somewhere a method is decorated with an attribute that triggers:

• https://stackoverflow.com/questions/25153974/customattributeformatexception-binary-format-of-the-specified-custom-attribute or
• https://stackoverflow.com/questions/26686523/bizarre-behavior-with-custom-attributes-and-getcustomattributes

but this one could be a long shot, there are very few reports of this over a very long time frame.

In either case, unfortunately, there's a very strong (nearly 100%) probability that your deployment is broken (either because of a misconfiguration, or a tooling bug), and it's only now being revealed because the change linked by @skomis-mm forces slightly more assembly metadata to be loaded than the earlier version did.

Definitely still open to helping here, but just letting you know why it's not something we're likely to find/reproduce outside your environments, so a lot more detailed info about those will be needed.

If you're super keen to track it down, breaking on all exceptions in a debugger and examining the attribute type that's being inspected on the line linked above should provide some useful clues.

Hope this helps!

[nunorelvao]

Hi, thanks for the reply, indeed i also was thinking the problem probably could be the target framework version being lesser than the local installed one.
Unfortunately I cannot debug more on the cloud as it is a corp controlled environment and lack some authorizations, but the fix by downgrading the package did the trick.

I understand the change introduced and on my case i doubt the reflection methods are picking any attributes causing issues on my minimal app code, yet it can also be because of some code on the framework itself..
As for build and deloyment we use the same building and target docker image with same framework version, hence this is is even stranger.