Add a way to determine whether an index setting is private from the API

[dakrone]

Currently we have certain index settings that use the Property.PrivateIndex property, and are thus unsettable from the regular REST API.

We should have a way to determine whether a setting is private from the API. For instance, when re-indexing an index, a user may want to copy the index's settings, but can't copy any private settings, or else the index creation will fail with "Validation Failed: 1: private index setting [index.frozen] can not be set explicitly"

[elasticmachine]

Pinging @elastic/es-core-infra

[dakrone]

We discussed this and didn't come to a complete decision, there were a few points that were brought up:

• We might want to whitelist settings that can be copied
• There are still multiple settings that are functionally private but not yet marked as such

As for solutions, there were a couple brainstormed:

• Provide a way to filter out private settings so only copy-able settings are seen from an API call
• Add a way to "clone" an index internal to ES that does the copy for you (for instance, for setting up for reindexing)

More discussion on this is needed.

[rjernst]

Private settings seem to me like an abuse of settings altogether. Settings, by definition, should be something that can be "set". Instead, private settings seem like metadata about the index that we store in settings so that we don't have to have separate index metadata. Perhaps we could add such metadata, migrate private settings to that, and deprecate and remove private settings altogether?

[dimitris-athanasiou]

I have also stumbled upon this trying to copy index settings during reindexing. I wonder if even better than determining private settings would be an API that returns the settings in a format that they can then be used during an index creation so that the new index has the same settings.

[jasontedor]

@dimitris-athanasiou We are discussing an idea like that in another channel indeed. The idea would be that we would make it easier to prepare an index for reindex by adding an API that could create an appropriate destination reindex target. That API would handle these concerns, without a client having to do things like hit an API to find the private settings, and then invoke the create index API with the appropriate settings, etc. There is no need to burden every caller with these concerns. How would a user even know to do such a dance? Instead, we can address these usability aspects by adding a dedicated API for reindexing workloads, then all of our users benefit, not only those that go through Kibana.

[dimitris-athanasiou]

@jasontedor That would be fantastic!