max_analyzed_offset warnings in header part of ES response #28830
Description
Elasticsearch version (bin/elasticsearch --version):
6.2.2
JVM version (java -version):
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (IcedTea 3.6.0) (Alpine 8.151.12-r0)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
OS version (uname -a if on a Unix-like system):
Linux es-client-5bdb9dc9dc-n4kfh 4.14.11-coreos #1 SMP Fri Jan 5 11:00:14 UTC 2018 x86_64 Linux
alpine container from https://quay.io/repository/pires/docker-elasticsearch-kubernetes running in kubernetes cluster behind nginx-ingress-controller.
Description of the problem including expected versus actual behavior:
max_analyzed_offset warnings in ES 6.2 produces response with unreasonably big headers - breaks Kibana deployed behind nginx-ingress-controller.
I'm using Kibana search and receiving BAD GATEWAY error.
Error Discover: Bad Gateway
Error: Bad Gateway
at respond (http://kibana-backend-stg.***/bundles/vendors.bundle.js?v=16588:111:161556)
at checkRespForFailure (http://kibana-backend-stg.***/bundles/vendors.bundle.js?v=16588:111:160796)
at http://kibana-backend-stg.***/bundles/vendors.bundle.js?v=16588:105:285566
at processQueue (http://kibana-backend-stg.***/bundles/vendors.bundle.js?v=16588:58:132456)
at http://kibana-backend-stg.***/bundles/vendors.bundle.js?v=16588:58:133349
at Scope.$digest (http://kibana-backend-stg.***/bundles/vendors.bundle.js?v=16588:58:144239)
at Scope.$apply (http://kibana-backend-stg.***/bundles/vendors.bundle.js?v=16588:58:147018)
at done (http://kibana-backend-stg.***/bundles/vendors.bundle.js?v=16588:58:100026)
at completeRequest (http://kibana-backend-stg.***/bundles/vendors.bundle.js?v=16588:58:104697)
at XMLHttpRequest.xhr.onload (http://kibana-backend-stg.***/bundles/vendors.bundle.js?v=16588:58:105435)
I've investigated the cause and it turns out that nginx rejects response from ES due to huge headers from ES:
nginx-ingress-controller-7669c9c9c7-c2h89 nginx-ingress-controller 2018/02/26 16:07:38 [error] 32351#32351: *5136430 upstream sent too big header while reading response header from upstream, client: 10.***.72, server: es-backend-stg.***, request: "POST /_search HTTP/1.1", upstream: "http://10.***.75:9200/_search", host: "es-backend-stg.***"
nginx-ingress-controller-7669c9c9c7-c2h89 nginx-ingress-controller 2018/02/26 16:07:39 [error] 32351#32351: *5136436 upstream sent too big header while reading response header from upstream, client: 10.***.72, server: es-backend-stg.***, request: "POST /_search HTTP/1.1", upstream: "http://10.***.89:9200/_search", host: "es-backend-stg.***"
tcpdump from client showed that ES dumps lots of highlighting warnings into a response before headers:
HTTP/1.1 200 OK [290/46590]
Warning: 299 Elasticsearch-6.2.2-10b1edd "The length of text to be analyzed for highlighting [86432] exceeded the allowed maximum of [10000] set for the next major Elastic version. For large texts, indexing with offsets or term vectors is recommended!" "Mon, 26 Feb 2018 16:13:44 GMT"
Warning: 299 Elasticsearch-6.2.2-10b1edd "The length of text to be analyzed for highlighting [10307] exceeded the allowed maximum of [10000] set for the next major Elastic version. For large texts, indexing with offsets or term vectors is recommended!" "Mon, 26 Feb 2018 16:13:44 GMT"
.....40x.....
content-type: application/json; charset=UTF-8
ES query which is issued by kibana (formatted):
POST /_msearch HTTP/1.1
{"index":["kafka-*"],"ignore_unavailable":true,"preference":1519659811998}
{
"version": true,
"size": 500,
"sort": [
{
"kafka_timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"_source": {
"excludes": []
},
"aggs": {
"2": {
"date_histogram": {
"field": "kafka_timestamp",
"interval": "1h",
"time_zone": "Europe/Berlin",
"min_doc_count": 1
}
}
},
"stored_fields": [
"*"
],
"script_fields": {},
"docvalue_fields": [
"@timestamp",
"kafka_timestamp"
],
"query": {
"bool": {
"must": [
{
"match_all": {}
},
{
"range": {
"kafka_timestamp": {
"gte": 1519486913449,
"lte": 1519659713449,
"format": "epoch_millis"
}
}
}
]
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"*": {}
},
"fragment_size": 2147483647
}
}
I've fixed my case by issuing below request:
curl -X PUT http://es-backend-stg.***/kafka-*/_settings -d '{ "index.highlight.max_analyzed_offset": 100000}'
Expected behaviour:
I'm not sure is the issue should be fixed on ES or Kibana side.
From ES perspective maybe limiting warnings to reasonable size, or logging them instead of putting into response headers may help..
Moving them in body field instead of headers may simplify middlware configuration and visibility.
Maybe Kibana request is another point where this could be fixed if warning-in-header behavior is preferred.