Opening indices with broken index settings

[jasontedor]

Today we are lenient with index settings, we do the following things:

1. We will open an index even if it contains an index setting that we do not recognize because we do not do any index settings validation when opening an index.
2. We will upgrade an index and archive unrecognized settings; this is bad in the following scenario:
   a. a user installs a plugin that provides an index setting
   b. a user creates an index that uses that index setting
   c. a user stops Elasticsearch to upgrade, so removes the plugin
   d. the user upgrades Elasticsearch, and starts Elasticsearch not having reinstalled the plugin
   e. Elasticsearch will archive the index setting and happily open the index

This leniency is dangerous and we should stop.

To be fair, the reason for the existence of this leniency in the upgrade case is for when we first introduced strict settings, we knew that we would see unrecognized settings on indices coming from 2.x into 5.x. We are past that point now. It's time to reconsider this leniency.

Relates #26995

[jasontedor]

I opened #26955 to address the first case. I would like to have a broader discussion about this.

[jasontedor]

We are going to proceed with the following plan:

• remove the leniency on opening indices with unrecognized settings and instead such an index will be closed with the unrecognized settings archived
• we will not open any index with archived settings
• users can remove archived settings via the wildcard archived.*

[mayya-sharipova]

@jasontedor Jason, I was studying your corresponding PR: https://github.com/elastic/elasticsearch/pull/26995/files

and I don't understand that why when we create an IndexService in IndicesService.createIndexService, we don't validate archive settings as well. Current code:
https://github.com/elastic/elasticsearch/pull/26995/files#diff-97e5abcaadbacda4084c902214cd3654R437

indexScopedSettings.validate(indexMetaData.getSettings(), true, true, true); //ignoreArchiveSetting: true 

Setting ignoreArchiveSetting to false, will validate archive settings as well, and will ensure that we never open an index with archived settings. No?

Thanks in advance for clarification.