[Transform] Allow aligning checkpointing with date_histogram bucket boundaries

[hendrikmuhs]

spin-off from #61587

If a transform is used with date_histogram transform creates and updates intermediate buckets, because checkpointing is not aligned with the interval in a the date histogram. For usecases with lot's of data and/or low intervals, this creates extra load, because upserts (delete + create) are expensive.

If checkpointing was aligned with bucket boundaries, upserts could be avoided.

Pro's

• better performance, no deletes
• destination could be a data stream
• destination could be managed with ILM (no duplicates, see Continuous transform with date histogram generates duplicates after index rollover #61587)
• avoid confusion with intermediate results in upstream usage

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)

[przemekwitek]

In order to achieve BWC, we should add a per-transform setting that will enable/disable this optimization.
There are two questions that come to mind:

What should the setting name be?

Propositions are:

• (more generic) optimize_checkpoint_ranges, enable_checkpoint_ranges_optimization
• (more specific) align_checkpoint_ranges_with_date_histogram_intervals.

My proposition is to go with the more generic name, e.g.: enable_checkpoint_ranges_optimization

What should the setting default value be?

Propositions are:

Default value	BWC	Adoption
enabled from the beginning	yes	biggest, all the new transforms get the optimization
enabled starting 7.x where x > 15	yes
enabled starting 8.0	yes
disabled	yes	marginal, only for support cases

My proposition is to go with the first option, i.e.: use the optimization by default from the start (7.15).

Special care should be taken during transform update when the default value changes between versions. One transform should always use consistent value of this setting throughout its life.