Upgrading to 5.6 Review

[elasticmachine]

Original comment by @bleskes:

Since the feature has been originally implemented (LINK REDACTED , LINK REDACTED), things have changed slightly. I have recently spoken to @jaymode , @pickypg and @spinscale to document how things currently work. Here's a summary which we can use for future reference, plus some follow ups at the end of each section.

@jaymode, @pickypg and @spinscale - please read carefully and correct if needed.

Security

Native Realm (.security index)

• ES 5.6 has introduced a new field, which prevents writing until the field has been added (see below when). Until this is done the native realm is read-only (on new nodes) as dynamic fields are disabled.
• Once the cluster master moves to version 5.6, xpack.security.support.NativeRealmMigrator does the following:
  • the security code issues a PUT template to update the .security index template. It also updates the mapping of the existing index if exists.
  • some internal users are updated depending on ES version.
• MetaDataUpgrader is used to upgrade the index template as well.
• Users need to manually call the _xpack/migration/upgrade API to reindex and remove types

.audit-* indices

• The mapping never changed and thus there is no need to upgrade the template
• We rely on non-6.x compatible indices to fall out of the retention policy and thus have no upgrade scheme for these.

TODOs

• Document that the native realm is read only until the master has been upgraded.
• We need to choose who updates the .security index template. It should only be done in one place.
• We need better testing to make sure that .security keeps on working after users have manually used the _xpack/migration/upgrade API. Those don't exist now.
  • Also take into account the upgrade of an index that was created on ES < 5.6, and upgraded to 5.6 first.
• Test that a mixed cluster .security works in read-only mode
• Test that we can write new user credentials after the cluster has been upgraded

Watcher

Watch CRUD & Execution

• Adding a watch tries to use the new doc type. If this fails, it tries again using the pre-5.6 types. This seems to result in ugly log messages - see todo.
• Watch execution runs on the master only and thus has a clean transition between old and new.
• The .watch and .triggered-watches* indices are manually upgraded via the _xpack/migration/upgrade API.

Watch history

• Template is automatically updated by the TemplateUpgradeService service.
• Since the watch execution is on the master, we only use the new template once the master moves to a new version.

TODOs

• We currently have the follow logs being logged until the user manually upgrades the .watch index. We should find a way to avoid it:

[2017-10-06T06:42:14,598][DEBUG][o.e.a.b.TransportShardBulkAction] [.watches][0] failed to execute bulk item (index) BulkShardRequest [[.watches][0]] containing [index {[.watches][doc][wAuN5cXhTiyjyCm58tH6ag_xpack_license_expiration], source[n/a, actual length: [4.5kb], max length: 2kb]}] and a refresh
org.elasticsearch.indices.TypeMissingException: type[doc] missing
        at org.elasticsearch.index.mapper.MapperService.documentMapperWithAu

• Delay watch execution on the master until the required template version is visible in the cluster state. (LINK REDACTED)

Monitoring

Exporting

Local exporter:

• The monitoring indices template is upgraded by the monitoring service, when the master moves to 5.6.
• The exporter waits until it sees the new template in place (i.e., until the master is on 5.6)

Http Exporter:

• Always tries to update the remote template when it sets up.

Monitoring indices

• ES 5.6 uses a new index name schemes - i.e., new indices will be created next to the old indices as soon as the first data is shipped.
• Old indices are not upgraded, we let them age and fall out of the retention policy.

TODOs

• Move template upgrades to the centralized TemplateUpgradeService. Beats will takeover this responsibility for Monitoring.
• Research the following log messages that repeated appear in the logs until the upgrade is complete

13:32:43 [2017-10-04T15:32:20,567][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [node-0] collector [cluster_stats] failed to collect data
13:32:43 java.lang.IllegalStateException: Security index is not on the current version - the native realm will not be operational until the upgrade API is run on the security index

and

[2017-10-06T06:42:14,621][ERROR][o.e.x.m.e.l.LocalExporter] failed to set monitoring watch [wAuN5cXhTiyjyCm58tH6ag_elasticsearch_version_mismatch]
org.elasticsearch.indices.TypeMissingException: type[doc] missing
        at org.elasticsearch.index.mapper.MapperService.documentMapperWithAutoCreate(MapperService.java:765) ~[elasticsearch-6.1.0-SNAPSHOT.jar:6.1.0-SNAPSHOT]
        at org.elasticsearch.index.shard.IndexShard.docMapper(IndexShard.java:2147) ~[elasticsearch-6.1.0-SNAPSHOT.jar:6.1.0-SNAPSHOT]
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:677) ~[elasticsearch-6.1.0-SNAPSHOT.jar:6.1.0-SNAPSHOT]

TemplateUpgradeService

The service is in charge of checking index template versions and upgrading them if needed. It currently tries to do so when the first 5.6 node joins the cluster. This fails because the _system user only has permissions to do so once the master has moved to 5.6 node (see LINK REDACTED). This results in repeated ugly messages about the _system not having the right permissions. Given the review of the different features above, we're safe to move to a simpler model where the templates are updated only on the current master (i.e., when the master is on the 5.6 version).

• Change the TemplateUpgradeService to only run its upgrades once the local node is master ( TemplateUpgradeService should only run on the master #27294 )

[elasticmachine]

Original comment by @spinscale:

@bleskes I was not able to reproduce the exception with a local test, maybe you can link to a test run here

In addition you started a sentence above with Delay watch execution on the without finishing it, please do so (I suppose you wanted to make sure I read the paragraph :-)

[elasticmachine]

Original comment by @bleskes:

@spinscale I updated the "delay watch sentence" sentence. I'll reach out about the log.

[elasticmachine]

Original comment by @jaymode:

ES 5.6 has introduced a new field, which prevents writing until the field has been added (see below when). Until this is done the native realm is read only (on new nodes) as dynamic fields are disabled.

For clarity and correctness, 5.6 didn't introduce a new field; but one could have been introduced between the previous version and current (i.e. 5.5 adding role mappings).

[elasticmachine]

Original comment by @spinscale:

I have spent some more time digging into the above mentioned log message flood and was unable to reproduce, coming from 5.5 to 5.6. Looking closer at your above log message I am confused now, as the message stack traces contain elasticsearch-6.1.0-SNAPSHOT.jar

[elasticmachine]

Original comment by @spinscale:

this was my test setup

# store ten of those my_watch1-10
for i in 0 1 2 3 4 5 6 7 8 9 ; do 

curl -u elastic:changeme -XPUT localhost:9200/_xpack/watcher/watch/my-watch_$i -d '
{
  "trigger": {
    "schedule": {
      "interval": "15s"
    }
  },
  "input": {
    "simple": {
      "foo": "bar"
    }
  },
  "actions": {
    "logging" : {
      "logging" : {
        "text" : "{{ctx}}"
      }
    }
    
  }
}'

done

Different setups

• minimum master nodes:1, three nodes total, security disabled
• minimum master nodes:2, three nodes total, security disabled
• minimum master nodes:1, three nodes total, security enabled

1. one data node, two dedicated masters on 5.5.3
2. store 10 watches, each running every 15 seconds
3. bring down the master eligible node which is not master and replace with 5.6.3
4. do the same with the master node
5. upgrade the data node
6. run upgrade API

# see that .watches and .triggered_watches indices are mentioned here
GET /_xpack/migration/assistance

POST _xpack/watcher/_stop

POST /_xpack/migration/upgrade/.watches

POST /_xpack/migration/upgrade/.triggered_watches

POST _xpack/watcher/_start

GET _xpack/watcher/stats

[elasticmachine]

Original comment by @bleskes:

I opened #27294 to deal with the point about the TemplateUpgradeService

[elasticmachine]

Original comment by @bleskes:

#27294 was merged and backportted to 5.6

[elasticmachine]

Original comment by @sophiec20:

FF tomorrow. Please confirm if this is a blocker for 5.6.8.

[elasticmachine]

Original comment by @jaymode:

I do not believe this is really a blocker

[elasticmachine]

Original comment by @bleskes:

@sophiec20 we definitely didn't treat as such since the initial push. I'll remove the blocker label (thanks for the ping). I'll also review what's left to do.

[elasticmachine]

Original comment by @pickypg:

Looking back at this issue, I have never really reproduced the security index issue (nor heard user complaints about it), which I attribute to a temporary race condition of the upgrade that just means that that monitoring data misses the cluster stats.

As for the doc _type issue, I did finally have it happen similarly. Basically the index had not yet been upgraded, so it was just another race condition. The next execution of it will presumably fix it (after they have fixed the index).

I was putting off the template upgrade service component until implementing the _xpack/monitoring/_setup endpoint, but that is no longer necessary as we plan to implement Beats modules instead. I'll see if it's easy to trigger, but I suspect the HTTP Exporter will not easily be able to upgrade the monitoring templates (and you don't want to do it automatically on the other side).

[elasticmachine]

Original comment by @spinscale:

are there still leftovers, that make this issue valid to be opened up? If so, would it make more sense to create dedicated issues?