HttpLogging Middleware option to log all headers

[liguori]

Background and Motivation

By default request/response headers which are not added to the collections of the RequestHeaders and ResponseHeaders properties of the HttpLoggingOptions configuration object are logged with a redacted value.

It would be good to add an option, in scenarios controlled by the developer, to enable the logging of any request or response header.

Examples of scenarios:

• Debugging
• Sensitive data by design of the application never ends up in the headers
• Not predictable headers that must be logged

Proposed API

It can be achieved by allowing new values for the HttpLoggingFields enum. For example:

namespace Microsoft.AspNetCore.HttpLogging;

[Flags]
public enum HttpLoggingFields : long
{
   //Existent enum values
   None = 0x0,
   RequestPath = 0x1,
   ....
+  RequestHeadersIncludeSensitive = 0x1000,
+  ResponseHeadersIncludeSensitive = 0x2000,
+  RequestPropertiesAndHeadersIncludeSensitive = RequestProperties | RequestHeadersIncludeSensitive,
+  ResponsePropertiesAndHeadersIncludeSensitive = ResponseStatusCode | ResponseHeadersIncludeSensitive,
+  RequestIncludeSensitive = RequestPropertiesAndHeadersIncludeSensitive | RequestBody,
+  ResponseIncludeSensitive = ResponseStatusCode | ResponseHeadersIncludeSensitive | ResponseBody,
+  AllIncludeSensitive = RequestIncludeSensitive | ResponseIncludeSensitive
   ....
   All = Request | Response
}

Usage Examples

The developer can opt-in for the new beahvior:

using Microsoft.AspNetCore.HttpLogging;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHttpLogging(logging =>
{
    logging.LoggingFields = HttpLoggingFields.AllIncludeSensitive; //Usage of one of the new enum values
});
var app = builder.Build();
app.UseHttpLogging();
app.Run();

This configuration will log all the request/response headers without the redacted value.

Risks

It doesn't affect nor change the current API behavior since the developer must explicitly configure the logging with one of the new values in order to enable the logging for all the headers.

Implementation

The following commit shows how could look like the implementation: liguori@9f35fa9

[blowdart]

Given that you can add to the RequestHeaders and ResponseHeaders in the middleware options why isn't that a sufficient balance between flexibility and security?

The problem I see here is what is Sensitive is context specific. It can and does change, not only based on your application, but on changes to the HTTP specs, or browser implementations.

As the .net security person going for a "log everything" presents an unacceptable risk for the vast majority of users. It's the same reason we switched from detailed exception messages by default, people would turn details on "for debugging" and forget, leading to information leaks at best and EoP at worse.

[liguori]

There are context in which headers are not predictable because dynamics. I am thinking about "X-" prefixed header that can arrive on the application depending on any intermediate infrastructural objects or cases in which you would like to log "abusive user/client behavior" that can inject every kind of "garbage" headers and it must be logged for auditing purpose.

In order to be still secure, when those new Enum values are set, it could be added an excluson list of well-known standard headers (from HTTP specs) in some new properties RequestExcluedHeaders and ResponseExcluedHeaders of the HttpLoggingOptions configuration object and initialized by default from the framework (i.e. Authentication, Cookie , etc), still giving the user choice to remove it.

The point is to be secure by design because default framework initialization but give the user (developer) the choice.

Another consideration is that the most of the PII or sensitive data is in the body and once you decide to enable logging with HttpLoggingFields.All or HttpLoggingFields.RequestBody (current possible configuration), you are logging everything.

[flucianomsft]

The problem I see here is what is Sensitive is context specific. It can and does change, not only based on your application, but on changes to the HTTP specs, or browser implementations.

100% agree @blowdart, however, in my opinion logging request/response body already exposes the most sensitive information (at least in the majority of use cases).

Also, if users are willing to accept the risk of the behavior, that could enable several real-world use cases, e.g.:

• asynchronous requests replay/traffic mirroring
• near-real time security analysis (think of a modsec scanning in detection mode performed asynchronously and without impacting hot paths)
• auditing in very specific use-cases

[blowdart]

Honestly, I look at your suggested use cases and that all suggest specialized middleware rather than attempting to hang off the back of the logging middleware. We (the ASP.NET team) have a responsibility to design for security by default and make it a conscious effort for users to code out of that secure by default. Giving a switch to log everything, from auth headers, through PII in form submissions doesn't meet that secure by default bar.

[Tratcher]

Prior request: #32246

[adityamandaleeka]

Triage: we like the approach in #39200 better for this sort of thing.

[liguori]

Thank you @Tratcher for pointing to the #33346, your suggestion about expose a boolean switch would fit the example scenarios that I have mentioned in the first message and that I would like to be covered (not only debugging).

I agree with @blowdart as well about the responsibility of security decisions of the ASP.NET team but I think that currently are already shipped with the logging middleware options for log information that could be potentially more sensitive than what we could find in all the headers (except some few exceptions similar to the one I have mentioned) like the Body that if we think about Rest API could expose all the possible data of this world.

@adityamandaleeka it is fine to close this issues I will follow the #39200 although I cannot figure out which selector could be used in order to enable in future the logging of ALL the headers (with the possibility to exclude someone like Authorization, Cookie, etc).