Counter field validation still occurs after downgrading tsdb data stream

[martijnvg]

Tsdb is more picky about what aggregations are allowed to be executed on counter fields. For example a sum aggregation isn't allowed to run on a counter field. There is support to opt out of tsdb. However even after opting out of tsdb shard failures may still occur. This is because the data stream is downgraded but not any pre-existing tsdb indices.

Reproduction:

PUT _index_template/1
{
  "index_patterns": [
    "test*"
  ],
  "template": {
    "mappings": {
      "properties": {
          "my_field": {
              "time_series_dimension": true,
              "type": "keyword"
          },
          "another_field": {
            "type": "long",
            "time_series_metric": "counter"
          }
      }
    }
  },
  "data_stream": {}
}


POST test2/_doc?refresh
{
  "@timestamp": "2023-05-16T11:49:50.599Z",
  "my_field": "value",
  "another_field": 656
}

GET test2/_search
{
    "size": 0,
    "aggs": {
        "test": {
            "sum": {
                "field": "another_field"
            }
        }
    }
}


PUT _index_template/1
{
  "index_patterns": [
    "test*"
  ],
  "template": {
    "settings": {
      "index": {
        "mode": "time_series"
      }
    },
    "mappings": {
      "properties": {
          "my_field": {
              "time_series_dimension": true,
              "type": "keyword"
          },
          "another_field": {
            "type": "long",
            "time_series_metric": "counter"
          }
      }
    }
  },
  "data_stream": {}
}

POST test2/_rollover

POST test2/_doc?refresh
{
  "@timestamp": "2023-05-16T16:23:51.659Z",
  "my_field": "value",
  "another_field": 6566
}

GET test2/_search
{
    "size": 0,
    "aggs": {
        "test": {
            "sum": {
                "field": "another_field"
            }
        }
    }
}

PUT _index_template/1
{
  "index_patterns": [
    "test*"
  ],
  "template": {
    "settings": {
      "index": {
        "mode": null
      }
    },
    "mappings": {
      "properties": {
          "my_field": {
              "time_series_dimension": true,
              "type": "keyword"
          },
          "another_field": {
            "type": "long",
            "time_series_metric": "counter"
          }
      }
    }
  },
  "data_stream": {}
}

POST test2/_rollover

POST test2/_doc?refresh
{
  "@timestamp": "2023-05-16T16:23:51.659Z",
  "my_field": "value",
  "another_field": 6566
}

GET test2/_search
{
    "size": 0,
    "aggs": {
        "test": {
            "sum": {
                "field": "another_field"
            }
        }
    }
}

The expected behaviour is that the last search succeeds but it fails with a shard failure:

{
    "took": 4,
    "timed_out": false,
    "_shards": {
        "total": 3,
        "successful": 2,
        "skipped": 0,
        "failed": 1,
        "failures": [
            {
                "shard": 0,
                "index": ".ds-test2-2023.05.16-000002",
                "node": "e9y5C7zmRy6I-3j5aW1vEA",
                "reason": {
                    "type": "illegal_argument_exception",
                    "reason": "Field [another_field] of type [long] is not supported for aggregation [sum]"
                }
            }
        ]
    },
    "hits": {
        "total": {
            "value": 2,
            "relation": "eq"
        },
        "max_score": null,
        "hits": []
    },
    "aggregations": {
        "test": {
            "value": 7222
        }
    }
}

Note that this shard failure could be ignored.

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[ruflin]

For all the "valid" aggregation, what happens with the tsdb indices after rollback? Assuming I look at 24h and after 12h I had a TSDB index for 1h and then rolled it back, will my aggs over the 24h show the correct results or will be there a missing entry for 1h? This is under the assumption, I use a max agg on a field that was a counter for 1h.

[martijnvg]

The max aggregation is supported on counter fields. And like any other supported aggregation for counter fields these will continue to work on tsdb data streams and downgraded data streams.

But for let's say a sum aggregation (and other unsupported aggs), which isn't supported aggregations on counter fields. These will not work on a tsdb data stream. And on a downgraded data stream, these aggregations will fail on backing indices that were created at the time the data stream was still in tsdb mode. This means no data/results will be returned for any searches have an unsupported aggregation in the time period a data stream was in tsdb mode. This will be returned as a shard error. Data/results will be returned from the time the data stream was downgraded to be a normal data stream.

[felixbarny]

Data/results will be returned from the time the data stream was downgraded to be a normal data stream.

That sounds good to me. Just to double-check that I understood this correctly: Say you upgrade a data stream with existing metric data to TSDB, then downgrade after 1h and continue ingesting data. If you do a sum aggregation on a counter, the response will only miss data from that 1h period where the data stream was in TSDB mode. Is that correct?

[martijnvg]

Say you upgrade a data stream with existing metric data to TSDB, then downgrade after 1h and continue ingesting data. If you do a sum aggregation on a counter, the response will only miss data from that 1h period where the data stream was in TSDB mode. Is that correct?

Yes. Additionally the search response will also report shard(s) failures, because sum aggregation was attempted on counter field in a tsdb backing index.

[felixbarny]

Is there anything we need to change then?

Note that this shard failure could be ignored.

We could ignore it but wouldn't that be even more surprising to users as there wouldn't be an explanation of why there's partially missing data?

[martijnvg]

Is there anything we need to change then?

I don't think so.

We could ignore it but wouldn't that be even more surprising to users as there wouldn't be an explanation of why there's partially missing data?

Yes, the shard level error should be returned.

[felixbarny]

@ruflin @lalit-satapathy do you agree that this is a non issue and that we should close this?

[lalit-satapathy]

@felixbarny the end user case is explained the duplicate here issue here, need to reproduce and verify once.

[ruflin]

The key question for me is how Lens / Kibana behaves:

1. No error should be thrown
2. Ideally we would inform the user why there is a gap (enhancement)