kibana/x-pack on tribe node issue

[obudiman]

Elasticsearch version: 5.0.0-alpha5

Plugins installed: x-pack@5.0.0-alpha5

JVM version: java-1.8.0-openjdk-1.8.0.101-3.b13.24.amzn1.x86_64

OS version: Linux 4.4.11-23.53.amzn1.x86_64

Description of the problem including expected versus actual behavior:
Kibana not usable when installed on tribe node with X-pack. We have followed the workaround as suggested on github #3114, but when trying to start the kibana, we found the following error messages on kibana log:

{"type":"log","@timestamp":"2016-09-19T06:57:33Z","tags":["status","plugin:xpack_main@5.0.0-alpha5-SNAPSHOT","error"],"pid":25694,"state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2016-09-19T06:57:33Z","tags":["status","plugin:graph@5.0.0-alpha5-SNAPSHOT","error"],"pid":25694,"state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2016-09-19T06:57:33Z","tags":["status","plugin:reporting@5.0.0-alpha5-SNAPSHOT","error"],"pid":25694,"state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2016-09-19T06:57:33Z","tags":["status","plugin:security@5.0.0-alpha5-SNAPSHOT","error"],"pid":25694,"state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2016-09-19T06:57:43Z","tags":["error","monitoring-ui"],"pid":25694,"level":"error","message":"Request Timeout after 30000ms","error":{"message":"Request Timeout after 30000ms","name":"Error","stack":"Error: Request Timeout after 30000ms\n    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:336:15\n    at [object Object].<anonymous    (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:365:7)\n    at Timer.listOnTimeout (timers.js:92:15)"}}

Looking further on the ES log, I can see the following error messages which seems to indicate it tries to create those indices from the tribe node:

[2016-09-19 16:58:33,210][DEBUG][action.admin.indices.create] [dcmarhel6d] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2016-09-19 16:58:33,210][DEBUG][action.admin.indices.create] [dcmarhel6d] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2016-09-19 16:58:33,239][INFO ][xpack.security.audit.logfile] [transport] [access_granted] origin_type=[rest], origin_address=[127.0.0.1], principal=[dcma.appservice], action=[indices:data/write/bulk]
[2016-09-19 16:58:33,239][INFO ][xpack.security.audit.logfile] [transport] [access_granted] origin_type=[rest], origin_address=[127.0.0.1], principal=[dcma.appservice], action=[indices:admin/create], indices=[_data]
[2016-09-19 16:58:33,240][DEBUG][action.admin.indices.create] [dcmarhel6d] no known master node, scheduling a retry
[2016-09-19 16:58:33,240][INFO ][xpack.security.audit.logfile] [transport] [access_granted] origin_type=[rest], origin_address=[127.0.0.1], principal=[dcma.appservice], action=[indices:admin/create], indices=[_xpack]
[2016-09-19 16:58:33,240][DEBUG][action.admin.indices.create] [dcmarhel6d] no known master node, scheduling a retry

We've tried to set the tribe node to be a data node to see if it will create the indices, but it didn't work although the kibana would start just fine. After reverting the data node to be a tribe node, we still found the same error messages.

Steps to reproduce:

1. Install kibana on tribe node
2. Follow workaround as stated on github return term vectors and some statistics for a document #3114 (pointing kibana to elected master node and create dashboard,search,visualization before repointing it back to tribe node)
3. Check kibana.stdout and elasticsearch.log

[clintongormley]

Kibana-with-tribe-nodes is not supported because of exactly the reasons you describe. Tribe nodes cannot be used to create indices. It looks like the current Kibana workaround that you point to is insufficient for v5, maybe because of Kibana monitoring.

As a side note, it would be nice if the log message in ES would specify which index it was trying to create when it timed out.

@pickypg @epixa any thoughts?

[epixa]

With the traditional workaround in place, I'd definitely be interested in figuring out why Kibana still thinks the Kibana index doesn't exist and needs to be created in v5. I'll have to stand up a cluster at some point to look into that. The ES index it is trying to create would be helpful, so +1 to that.

[obudiman]

Just an update,

Kibana on tribe node seems to work fine as long as we don't have the x-pack plugin installed in kibana.

We've also tried to disable the kibana monitoring by adding these lines on all es nodes:

xpack.monitoring.enabled: false
xpack.monitoring.collection.indices: -.kibana

but we still get the same error messages.

On a side note, isn't the index name it tried to create already listed on the ES log?

... action=[indices:admin/create], indices=[_data]
... action=[indices:admin/create], indices=[_xpack]

[epixa]

@obudiman What happens if you try to disable reporting? xpack.reporting.enabled: false

[epixa]

The short of it is, Kibana creates more indices now than it used to, and it's not a once-off thing. The new monitoring capabilities in Kibana creates new indices daily, and the reporting system creates a new index every week. I'm not sure of any other situation off-hand where this occurs, but it's clear that the traditional workaround will not longer be sufficient.

[obudiman]

The option is not recognized in ES a5, but thanks for that and the info.

Just curious if there's any plan to get kibana to work with the tribe node, so that we can have one kibana which can do visualization across multiple ES clusters?

[jordansirwin]

This is a major concerns for us, as a customer. We have many clusters unified by tribe nodes, and Kibana provides a single pane of glass into that data. No longer being able to do that will completely block us from upgrading.

Why not provide separate configuration options for where Kibana should store its data vs. where it should be searching?

[obudiman]

Either that or maybe just provide an option to allow those internal indices to be written to a specific cluster as described in #15620.

[s1monw]

@epixa we discussed this during our weekly fix-it meeting and we agreed that this is a kibana issue rather than an issue in tribe after all? Kibana should somehow have the ability to connect to 2 clusters one for writing one for reading? I really don't think we should relax the limitation of tribe to be read-only. Is this something we can make configurable in kibana to have separate connections and make them the same unless tribe is configured?