Disable sort optimization in search_after and scroll requests

[dnhatn]

We can't enable the sort optimization with points in scroll requests until LUCENE-10119 is integrated.

I labelled this PR non-issue for an unreleased bug.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[jimczi]

Should we disable for all request with search_after and not only scroll ones ?

[dnhatn]

@jimczi I think search_after should be okay, but it's safer to disable them all now.

[dnhatn]

@jimczi I've disabled the optimization for search_after requests in 6d8ea42. Can you take another look? Thank you!

[jimczi]

LGTM, thanks @dnhatn

[mayya-sharipova]

Elasticsearch search_after is not affected by this, and I think we don't need to disable it.

The Lucene bug was for cases:

1. there is a single sort and
2. the exact FieldDoc from the previous page is provided with the exact Lucene docID, as we do for scroll scrollContext.lastEmittedDoc

For a general elasticsearch with search_after we do SearchAfterBuilder::buildFieldDoc and we either

1. don't know the exact docId field of FieldDoc
2. or if a user is using _doc or _shard_doc with the main sort, there will not be a single sort

[dnhatn]

@mayya-sharipova I agree with your analysis. @jimczi Do you still prefer to disable the sort optimization in search_after requests?

[jimczi]

or if a user is using _doc or _shard_doc with the main sort, there will not be a single sort

Not necessarily, _shard_doc can be used as the single sort in a PIT.

It's not really an issue anyway since we don't intend to release any version with this regression.

[mayya-sharipova]

Not necessarily, _shard_doc can be used as the single sort in a PIT.

I think sort optimization is only applied for long fields, and is not applicable if there is a single sort on _shard_doc and this sort does't use NumericComparator.

But I am also ok to disable sort optimization with search_after completely for now to be completely sure we don't access it wrongly somewhere else internally (_seq?)

[mayya-sharipova]

@dnhatn Thanks a lot.

[dnhatn]

@jimczi @mayya-sharipova Thanks for reviews.