Composite aggs should support missing

[hendrikmuhs]

Composite aggs allows the user to specify "missing_bucket":true which has been implemented for rollup to handle sparse data.

For transform it would be good to support missing in addition, see #48243

Comparison missing_bucket and missing

missing_bucket

• maps non-existing fields to a bucket with key null
• its not possible to collapse the bucket into the bucket of the same value, e.g. if you want a default

missing

• maps non-existing fields to a bucket with a given key, key must not be null
  • the key can clash on purpose with an existing bucket, so you can collapse buckets, e.g. "missing":0 in a histogram
• allowing "missing": null would be possible as it is valid json, however for high-level clients this is problematic as some default to remove the whole entry if the value is null, therefore it's better to keep disallow null and handle this specific case with "missing_bucket":true

Solution

Composite aggs should support missing(+ it would be good if aggregations support missing_bucket for feature parity).

From a user perspective the 2 parameters might cause some confusion, because it's not clear whether you have to set missing_bucket to true if you set missing. Therefore it seems best to ignore missing_bucket if missing is set or make both parameters mutually exclusive.

[markharwood]

A user reported today that while upgrading from 6.x to 7.x they noticed the missing parameter had gone missing.

It turns out this was a feature of the composite agg but was deprecated with this change. I don't know what the rationale was for removing, just adding a note here as it's likely relevant.

[markharwood]

I don't know what the rationale was for removing,

I spoke to Jim today. Apparently the reason for originally removing the missing option from composite was that there were complications related to roll ups involving multiple indices. Best to ask Jim for more details on that.

[wchaparro]

closing as not planned.