[ML] Make Updating Data Frame Transforms Easier

[andrewvc]

Currently updating a dataframe transform is cumbersome. One must:

1. Check that it exists
2. Check to see if it is in a started state
   1. If it is started, it must be stopped
3. The existing dataframe transform must be deleted
4. A new one must be installed
5. The new one must be started.

It would be preferable to have all this happen behind the scenes and allow users to simply PUT their new dataframe over the old one. Doing so would internally perform the steps above.

[elasticmachine]

Pinging @elastic/ml-core

[benwtrent]

I do agree that we should have better facilities for updating data frame transforms. What fields are able to be updated is up for debate.

• Should we allow updating the group_by even though that changes the document ID generation?
• Should we allow the aggregations to be updated?
• Updating aggs or the group_by also implies that the mapping of the destination index is now wrong. Should we attempt to create the appropriate mapping and add fields?

Fields that I think are pretty safe to update:

• source <- including the query and index
• dest <- including index and the newly added pipeline field
• description
• pivot.max_page_search_size

I do not think that the update action should stop and then start the transform again. We should probably only allow updating a stopped data frame. Calling _stop against an already stopped data frame is a no-op.

[andrewvc]

I don't anticipate for our use case needing to update any fields other than perhaps pivot.aggregations and description.

That said, let's imagine a future where we wanted to update the group_by definition to add a field. We could do it seamlessly by simply adding a version field to the documents in the destination index, and only querying documents from the latest run (or, alternatively, deleting_by_query all the previous versions).

Speaking of versioning, I'm considering using the description field to version our dataframe. On each beat startup we want to ensure the last dataframe is running, and that's the only real metadata field available. The alternative is just to recreate the frame each time, but at scale that's a problem.