Add HTTP trailers to chunked REST infrastructure

[original-brownbear]

Follow up to #88311:

some REST responses will generate headers (e.g. deprecation headers) during serialization. We can't add them as actual headers in the chunked encoding approach since we don't know them until we finish the serialization. We can however add them as trailers after serialization. I'll implement that here.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

@sethmlarson says:

I would avoid using trailers if possible, many HTTP libraries don't process them or make them available to users so wouldn't be actionable from clients team.

I see a few options for handling response headers added after we've started sending the HTTP body:

1. Silently drop late-arriving deprecations and other headers. We do not require clients to process deprecation metadata in responses, and we don't use response headers for anything else today, so it's not a big loss in practice.

2. Forbid late-arriving headers (e.g. assert they don't occur). This would mean we don't miss anything, at the cost of having to implement some kind of pre-serialisation check for extra headers.

3. Add them as HTTP trailers. This is a valid (if obscure) part of the core HTTP spec and has been for >20 years so if they're not supported by a particular HTTP library then this seems like a library bug.

Possibly we need a blend of these options. Deprecation warnings are kinda safe to drop because the information also exists in the deprecation log, but dropping other response headers might be a trap for some future feature development so I would prefer to be stricter. If we use HTTP trailers then we must emit a Trailers: header listing the possible trailer names, but in theory we can add arbitrary response headers later on so we'd have to drop or forbid any ones that were not mentioned up-front.

Is this actually a problem today? If not, maybe we should just start by forbidding everything and deal with exceptions on a case-by-case basis.

I think this deserves discussion by the core/infra folks.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)