Send HTTP Warning Header(s) for any Deprecation Usage from a REST request #17804

pickypg · 2016-04-16T01:03:05Z

This adds support for ThreadContexts to maintain unique Response headers (repeated header values are ignored) in addition to Request headers. In particular, this enables REST requests (e.g., PUT /test) that hop across the nodes to properly return headers as part of the overall request. With the response headers Thread aware thanks to the ThreadContext, we can associate individual requests with deprecation logging and therefore warn developers when they unwittingly use deprecated features:

PUT /test
{
  "mappings": {
    "type": {
      "properties": {
        "field": {
          "type": "string"
        },
        "field2": {
          "type": "string"
        }
      }
    }
  }
}

The headers that would get returned by this would be:

HTTP/1.1 200 OK
Warning: The [string] field is deprecated, please use [text] or [keyword] instead on [field]
Warning: The [string] field is deprecated, please use [text] or [keyword] instead on [field2]
Content-Type: application/json; charset=UTF-8
Content-Length: 21

Alongside that feature, this adds a DeprecationRestHandler that logs a deprecation warning whenever it is used, but it otherwise behaves identically to normal RestHandlers. This allows REST APIs to be deprecated in a consistent manner instead of dropped without any non-documentation-level deprecation warnings in advance.

Closes #17687

jasontedor · 2016-04-16T01:07:50Z

core/src/main/java/org/elasticsearch/common/Strings.java

I don't think this method belongs on Strings, especially since it's only used in only place. I'd be fine with it as a static package-private method on DeprecationRestChannel so it's visible for testing.

I honestly think that it should be used in other places.

That's probably correct, but right now it's not and I don't think Strings is the right place anyway.

Moved to DeprecationRestChannel and also added String requireValidHeader(String value) that behaves similar to Objects.requireNonNull to use for all creations. I did leave it as public as it's used by DeprecationRestHandler (even though they're in the same package).

jasontedor · 2016-04-16T01:09:07Z

core/src/main/java/org/elasticsearch/index/shard/IndexShard.java

This too is confusing.

jaymode · 2016-07-01T16:24:34Z

core/src/test/java/org/elasticsearch/rest/DeprecationHttpIT.java

can we add text here/open up a issue?

Created #19222 (and placed it appropriately)

pickypg · 2016-07-05T22:08:03Z

@jaymode Please take a second pass now that I have rebased it back onto master. I think it's ready to merge with all tests passing.

jaymode · 2016-07-06T11:56:16Z

core/src/main/java/org/elasticsearch/common/logging/DeprecationLogger.java

I think it would be simpler to have the node remove its threadcontext from here on close

jaymode · 2016-07-06T11:58:44Z

left one comment, other than that LGTM

This adds a new proxy for RestHandlers and RestControllers so that requests made to deprecated REST APIs can be automatically logged in the ES logs via the DeprecationLogger as well as via a "Warning" header (RFC-7234) for all responses.

frioux · 2018-01-17T19:42:34Z

In case anyone in the internet superfuture runs into this like we did, this can pretty easily cause a long enough set of headers that nginx cannot even parse the response, and thus can cause pretty serious outages. Would be great if it could be disabled entirely, but you can set proxy_buffer_size to something higher to resolve it at least temporarily.

jasontedor · 2018-01-18T04:48:21Z

@frioux Can you share the contents of these large responses? I would be interested in seeing if there are headers that we need to de-duplicate (we do this already to some extent, this is part of what pushes my interest here).

frioux · 2018-01-18T04:58:13Z

HTTP/1.1 200 OK
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
Warning: 299 Elasticsearch-5.4.0-780f8c4 "The [string] field is deprecated, please use [text] or [keyword] instead on [REDACTED]" "Wed, 17 Jan 2018 19:03:29 GMT"
content-type: application/json; charset=UTF-8
content-encoding: gzip
transfer-encoding: chunked

1

Of course instead of REDACTED it was a distinct field for each thing. 12k is a lot of data to jam into headers ;)

jasontedor · 2018-01-18T05:01:38Z

Okay; it is because it's a distinct field. Thanks for that, I will see what, if anything, can be done (I am doubtful because of the way these are collected).

frioux · 2018-01-18T05:08:21Z

It would be good to limit this, because this is pretty gnarly to debug (I could only do it with a network trace) and anyone who puts nginx in front of ES would run into it.

jasontedor · 2018-01-18T17:06:30Z

I agree with you. I opened #28301 and this work is assigned.

pickypg added >enhancement :Core/Infra/Logging Log management and logging utilities :Core/Infra/REST API REST infrastructure and utilities v2.4.0 v5.0.0-alpha2 v2.3.2 review labels Apr 16, 2016

pickypg force-pushed the feature/deprecated-rest-handler-17687 branch from 8235156 to f6e5a18 Compare April 16, 2016 01:05

jasontedor reviewed Apr 16, 2016
View reviewed changes

pickypg force-pushed the feature/deprecated-rest-handler-17687 branch from f6e5a18 to c1d086a Compare April 16, 2016 01:09

jasontedor reviewed Apr 16, 2016
View reviewed changes

core/src/main/java/org/elasticsearch/index/shard/IndexShard.java Outdated

Copy link

Member

jasontedor Apr 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This too is confusing.

jaymode reviewed Jul 1, 2016
View reviewed changes

pickypg mentioned this pull request Jul 1, 2016

Thread Context should collect unique response headers in async actions #19222

Closed

pickypg force-pushed the feature/deprecated-rest-handler-17687 branch 2 times, most recently from 4f44659 to af4fc01 Compare July 5, 2016 20:35

jaymode reviewed Jul 6, 2016
View reviewed changes

pickypg force-pushed the feature/deprecated-rest-handler-17687 branch from af4fc01 to 8079da3 Compare July 6, 2016 22:55

pickypg force-pushed the feature/deprecated-rest-handler-17687 branch from dd44506 to b927cfe Compare July 6, 2016 23:52

pickypg removed the review label Jul 6, 2016

pickypg merged commit b927cfe into elastic:master Jul 6, 2016

clintongormley added the release highlight label Jul 7, 2016

pickypg changed the title ~~Add DeprecationRestHandler to automatically log deprecated REST calls~~ Send HTTP Warning Header(s) for any Deprecation Usage from a REST request Jul 7, 2016

pickypg deleted the feature/deprecated-rest-handler-17687 branch July 7, 2016 17:34

djschny mentioned this pull request Jul 12, 2016

Change Deprecated REST Handler prefix to be more granular #19377

Closed

nik9000 mentioned this pull request Aug 3, 2016

Fail yaml and doc tests when they receive unexpected Warning header #19793

Merged

pickypg mentioned this pull request Aug 17, 2016

Move Writer interface to Writeable #20036

Merged

jasontedor mentioned this pull request Jan 18, 2018

Add cluster level setting to control the max size of warning headers in client HTTP responses #28301

Closed

olcbean mentioned this pull request Mar 8, 2018

REST : deprecate field_data for Clear Indices Cache API #28943

Merged

orangejulius mentioned this pull request Dec 5, 2018

Support Travis VM Infrastructure pelias/schema#337

Closed

jbudz mentioned this pull request Jan 30, 2019

Support for response warning header elastic/elasticsearch-js#762

Closed

lukasolson mentioned this pull request Nov 3, 2021

[data.search] Remove warning toast elastic/kibana#117252

Merged

9 tasks

Send HTTP Warning Header(s) for any Deprecation Usage from a REST request #17804

Send HTTP Warning Header(s) for any Deprecation Usage from a REST request #17804

Uh oh!

Conversation

pickypg commented Apr 16, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pickypg commented Jul 5, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaymode commented Jul 6, 2016

Uh oh!

frioux commented Jan 17, 2018

Uh oh!

jasontedor commented Jan 18, 2018

Uh oh!

frioux commented Jan 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasontedor commented Jan 18, 2018

Uh oh!

frioux commented Jan 18, 2018

Uh oh!

jasontedor commented Jan 18, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

pickypg commented Apr 16, 2016 •

edited

Loading

frioux commented Jan 18, 2018 •

edited

Loading