[Transforms] Add support for search-time runtime fields to transforms

[droberts195]

Since #64374 Elasticsearch has the option to have search-time runtime fields.

We should allow these to be used in transforms.

We may want to ban certain critical fields in the transform configuration from being runtime fields if having them unindexed will kill performance. (For example, fields that we need to sort on, or do terms searches efficiently on.)

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)

[hendrikmuhs]

We may want to ban certain critical fields in the transform configuration from being runtime fields

👍 I am concerned by fields in the group_by in a transform that runs in continuous mode. If I understand correctly a terms query on a runtime field would cause a full scan of the source.

@nik9000 is this correct?

However, we have the situation already and we do not ban certain configuration at the moment, but give a performance warning. The problem is similar to using a script in the group_by. The warning is created, if we do not find at least one non-scripted group_by.

In summary the problem applies to 3 cases:

• search based rt-fields
• mapping based rt-fields
• scripts in group_by

As stated, right now there is no ban, but a warning, I am open for discussing this. However banning rt-fields as group_by in general is in my opinion not required. If you have another non-scripted group_by performance is usually ok. I saw cases with 7 other group_by's and only the 8th was using a script.

[przemekwitek]

Search-time runtime fields are now supported in Transform.
They are never banned, even if performance would be affected. Audit warnings are emitted if all the group-by fields or a sync time field is a search-time runtime field.