[ML] Move find_file_structure endpoint to dedicated plugin #67001
Description
The find_file_structure endpoint is currently experimental and part of the ML plugin.
There is a plan to make greater use of the corresponding Kibana file upload functionality outside of the Kibana ML app. This means that it is no longer acceptable that the find_file_structure API might be arbitrarily disabled in Elasticsearch simply because the rest of ML has been disabled for some reason. To avoid this problem the find_file_structure API needs to be moved to a separate plugin that cannot be disabled.
A few decisions need to be made.
Firstly, what should the new plugin and endpoint names be? Perhaps the most obvious plugin name is file_structure. That would lead to requests like this:
POST _file_structure/find_file_structure
So maybe then it makes sense to shorten the endpoint name to just find_structure, leading to:
POST _file_structure/find_structure
But maybe there is a better idea.
Next, should we continue to handle the old route for a while? We don't have to, because the endpoint was experimental, but it would be friendlier to do so. So then until 7.last we'd have a preferred route of _file_structure/find_structure and a deprecated alternative of _ml/find_file_structure. This would mean file upload would work during rolling upgrades at the point where the Elasticsearch cluster was on the newer version but Kibana was still on the older version.
As mentioned before this new plugin should NOT have a setting to enable/disable it - it should always be enabled.
The docs also need moving. We need to consult with the ML docs team to decide how best to do this. I am not sure if there's a precedent for how to structure the docs of a plugin that only has one endpoint.
Finally, the endpoint should no longer be experimental, in the sense that we commit not to completely remove it in the future. However, we don't want this to be interpreted as meaning that we'll commit to making it work perfectly for every file in the world. So we probably need some other disclaimer to add to the new docs page. Something along the lines of, "This endpoint applies a variety of techniques to detect a configuration suitable for importing a range of file structures. However, this is not to say it will find the optimum configuration for importing every file structure that exists. If you have a file for which a suboptimal configuration is determined you are welcome to report this and we will consider making improvements to the endpoint, but we cannot guarantee to adjust the endpoint to handle every possible file in the way you would like."