Investigate severity of upgrading to 8.0.1 despite terminal deprecation of old features

[grcevski]

Elasticsearch Version

upgrade from 7.x -> 8.0.0

Installed Plugins

No response

Java Version

bundled

OS Version

Linux

Problem Description

Recently we encountered a user issue report where upgrades to 8.0.1 from 7.17.1 caused an unhealthy cluster because the option index.indexing.slowlog.level wasn't removed prior to upgrading.

We'll use this issue to understand how this happened and ways we can prevent the upgrade before irreversible changes are done to the Elasticsearch metadata or data.

Steps to Reproduce

• Get Elasticsearch 7.9 cluster with the index.indexing.slowlog.level option set.
• Upgrade to 7.17.1 ignore the deprecation warning
• Upgrade to 8.0.1

Expected behaviour: the 8.0.1 upgrade should fail to start but not upgrade any metadata or data
Current behaviour: the upgrade to 8.0.1 proceeds irreversibly modifying the metadata before options are validated and start is unsuccessful.

Logs (if relevant)

No response

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[pgomulka]

looks like indeed there is a bug.

I reproduced this on 7.17.1

PUT test2
{"settings": {

	"index.search.slowlog.threshold.query.warn": "100ms",
	"index.search.slowlog.threshold.query.info": "100ms",
		"index.search.slowlog.threshold.query.debug": "0ms",
	"index.search.slowlog.threshold.query.trace": 0,
	"index.indexing.slowlog.threshold.index.trace": 0,
	  "index.indexing.slowlog.source": 5,
	  "index.indexing.slowlog.level": "INFO"
}
}

#! [index.indexing.slowlog.level] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "test2"
}

Deprecation logs do show this warning as CRITICAL.

I can see in codebase it is correctly set as Property.Deprecated - not DepreactedWarning

elasticsearch/server/src/main/java/org/elasticsearch/index/IndexingSlowLog.java

Line 76 in bcfbf00

Property.Deprecated

In my view the problem with deprecation logs in upgrade assistant is that it does not look too encouraging to check them. There is a link and I am not sure many people will click it. Most likely people will just hit upgrade when there is nothing critical on the first page.

However the deprecation info api do not return the setting as critical

  GET /_migration/deprecations

  "index_settings" : {
    "test2" : [
      {
        "level" : "warning",
        "message" : "Setting [index.indexing.slowlog.level] is deprecated",
        "url" : "https://ela.st/es-deprecation-7-slowlog-settings",
        "details" : "Remove the [index.indexing.slowlog.level] setting. Use the [index.*.slowlog.threshold] settings to set the log levels.",
        "resolve_during_rolling_upgrade" : false,
        "_meta" : {
          "actions" : [
            {
              "action_type" : "remove_settings",
              "objects" : [
                "index.indexing.slowlog.level"
              ]
            }
          ]
        }
      }
    ]
  },

---

the problem is that the check uses incorrect level

elasticsearch/x-pack/plugin/deprecation/src/main/java/org/elasticsearch/xpack/deprecation/IndexDeprecationChecks.java

Line 603 in eed16a2

DeprecationIssue.Level.WARNING,

will look into a fix

[pcsanwald]

have we written an SKI for this, if not, can we please do that?

[pgomulka]

just a follow up. In 7.17.2-SNAPSHOT it works fine and upgrade assistant presents critical deprecation:

[pgomulka]

What surprises me is that when I try reproducing this on my 1node cluster, after upgrade index has status yellow and the level setting is archived (archived.index.indexing.slowlog.level)
However when I try this on cloud from 7.17.1, after upgrade index is red and the level is not archived..
@grcevski do you have any ideas?

[grcevski]

Oh, very interesting. Looking at the current code it looks like we should be archiving and it's what happens for me locally too. I wonder if we have fixed a bug since 8.0.1 that resolved this issue?