Aggregation to select top N buckets in a parent multi-bucket aggregation

[veesahni]

Describe the feature: Ability to retain only the top N buckets given an aggregated metric on parent buckets. This would be similar to the bucket_selector aggregation (which removes buckets based on a script).

Example: Given a filters aggregation that's provided with 300 filters, we have a sub-aggregation that calculates a metric. We only want the results of buckets having the 5 largest values of the aggregated metric. Today, without a "Top N bucket selector" aggregation, the client has to retrieve all 300 and sort/filter on client side.

[cyrilleverrier]

I am very interested in this feature.

Use case: considering many sensors that send a numeric metric (@timestamp + key/value pair)
I want to get a sorted list of the top 10 sensors based on their latest value by @timestamp

POST mybeat/metrics/
{
  "@timestamp": "2017-01-25T08:00:00.000Z",
  "sensorID": "Sensor1",
  "my_data": 10
}

POST mybeat/metrics/
{
  "@timestamp": "2017-01-25T08:00:00.000Z",
  "sensorID": "Sensor2",
  "my_data": 20
}

POST mybeat/metrics/
{
  "@timestamp": "2017-01-25T08:10:00.000Z",
  "sensorID": "Sensor1",
  "my_data": 1
}

POST mybeat/metrics/
{
  "@timestamp": "2017-01-25T08:10:00.000Z",
  "sensorID": "Sensor2",
  "my_data": 2
}

POST mybeat/metrics/_search?filter_path=aggregations
{
  "aggregations": {
    "BY_SENSOR": {
      "terms": {
        "field": "sensorID.keyword"
      },
      "aggregations": {
        "LATEST_TIMESTAMP": {
          "terms": {
            "field": "@timestamp",
            "order": {
              "_term": "desc"
            },
            "size": 1
          },
          "aggregations": {
            "LATEST_VALUE": {
              "avg": {
                "field": "my_data"
              }
            }
          }
        }
      }
    }
  }
}

The search returns the following result:

{
  "aggregations": {
    "BY_SENSOR": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Sensor1",
          "doc_count": 2,
          "LATEST_TIMESTAMP": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 1,
            "buckets": [
              {
                "key": 1485331800000,
                "key_as_string": "2017-01-25T08:10:00.000Z",
                "doc_count": 1,
                "LATEST_VALUE": {
                  "value": 1
                }
              }
            ]
          }
        },
        {
          "key": "Sensor2",
          "doc_count": 2,
          "LATEST_TIMESTAMP": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 1,
            "buckets": [
              {
                "key": 1485331800000,
                "key_as_string": "2017-01-25T08:10:00.000Z",
                "doc_count": 1,
                "LATEST_VALUE": {
                  "value": 2
                }
              }
            ]
          }
        }
      ]
    }
  }
}

I want to perform a "top_hits" aggregations on "BY_SENSOR>LATEST_TIMESTAMP>LATEST_VALUE".

Any know solution but to do it in the application code?

[markharwood]

@cyrilleverrier that sounds like a question for the discuss forums and not the same topic as the original question.

@veesahni are you still in need of this feature? We haven't come across many people with this sort of requirement. I'm not sure what the 300 filters represent and who maintains them but obviously if they were instead applied on ingest e.g. defined as tagging rules in percolator then you'd have tokens in your index which would be more efficient to aggregate on and you could use the terms aggregation with size 5 to prune the aggregation tree.
The "filters-trimming" feature sounds plausible but I'm just not sure how big the need is for this.

[veesahni]

@markharwood yes, still need this one.

We have a bayes classifier implemented on top of ES. We index a bunch of documents that contain a large text body field (i.e. lots of tokens) and other metadata fields that are used for filtering. Then, given N words (decided on the fly), the client retrieves the number of docs where the words appear in the body field (i.e. each turns into a separate term filter based agg and we extract the doc_count). It then throws away all except the top M (i.e. those with the highest doc counts) for further calculations.

A percolator would require every possible word to have a filter defined in advance (or on the fly) - don't think that would be practical.

A top-N filter would minimize what ES needs to stream between nodes and to the client. It would be a similar optimisation to how a top_hits works, but with a custom underlying calculation (i.e. agg).

[markharwood]

If you're doing this sort of text analysis have you considered using the significant text aggregation? Some of the features that can help:

1. Works with the sampler aggregation to analyze only the highest-scored docs.
2. Works with a diversified-sampler aggregation to avoid samples being skewed from any one source e.g. no more than N articles from any one news source
3. Can remove noise from duplicate sections of text from the statistical analysis
4. Returns per-term stats for both result set and corpus-level doc frequencies.

[veesahni]

A sampler of some kind could be used to speed things up at the cost of some accuracy.

We haven't experimented with the significant text aggregation. Although it's not directly helpful in this case, it's application would be interesting in building a classifier.

Thanks for the tips! Today, we keep things reasonably fast by limiting the docs that need to be queried. Ultimately, it would be ideal to be able to further speed things up while retaining accuracy. This is what the feature request would enable.

[markharwood]

I guess what's unusual about this scenario is that ordinarily the need to "trim" results is related to aggregations that promote discovery e.g. a terms agg on a field may be used to discover previously unknown actors (e.g. find the top 50 customer accounts) and it is necessary to trim the potentially millions of candidates.
In the case of the filters agg, the client must provide known filter definitions up-front and these definitely do not measure in the millions which is why we've not seen a big demand for pruning of these filter results. I'm trying to gauge if this request is more about reducing network overheads (how big are the results under each bucket?) or just simplifying your client's code.

cc @elastic/es-search-aggs

[veesahni]

I agree with your assessment as to why you don't see this requested much.

We're leveraging ES's parallel querying capabilities to implement an algorithm. The results under each bucket are minimal (we just need doc counts). The client code to handle the filtering isn't that complex (in a modern language, it doesn't take much to sort and filter hundreds of buckets). Accordingly, we implement the filtering in the client today. The added cost is some network overhead to bring the data to the client and all the processing done at the client location.

With the requested feature implemented, each ES node would need to only send it's top results, which would furthered be filtered before sending to the client. The computational effort of filtering would be spread across the physical nodes and network communication would also be minimised. Thus, my expectation is to see an overall improvement in query times.

[colings86]

This feature has been added in #27152 and is available in version 6.1 onwards