Move early termination based on index sort to TopDocs collector

[jimczi]

Lucene TopDocs collector are now able to early terminate the collection based on index sort (https://issues.apache.org/jira/browse/LUCENE-8059). This change plugs this new functionality directly in the query phase instead of relying on a dedicated early terminating sorting collector.

[jpountz]

Thanks for tackling this change. It took me some time to understand how it works (not your fault!) but it looks good to me in general. I left a suggestion about a potential improvement to the early-terminating logic.

[jpountz]

👍

[jpountz]

I could be wrong, but I think we could simplify it a bit by doing something like that:

// implicit total hit counts are valid only when there is no filter collector in the chain
int count = hasFilterCollector ? -1 : shortcutTotalHitCount(reader, query);
boolean doTrackTotalHits = trackTotalHits && count == -1; // we can also skip total hit counts if the query gives it for free
final TopDocsCollector<?> topDocsCollector = TopFieldCollector.create(sortAndFormats.sort, numHits,
    (FieldDoc) searchAfter, true, trackMaxScore, trackMaxScore, doTrackTotalHits);

This way, we don't need to check whether the collector can actually early-terminate or not, and we never have to add a TotalHitCount collector?

[jimczi]

We still want the top docs collector to early terminate if track_total_hits is true.
This is why we have the complex logic that wraps a counting collector in the next block.
We could also add a filtered collector that intercepts CollectionTerminatedException coming from the top docs collector and continue the collection with a simple counting collector to get the total hit count. This way we can always assume that the TopDocsCollector can early terminate ?

[jimczi]

I pushed 09a033a to simplify the logic. Can you take another look ?