Add search_after parameter to new terms_enum api

[markharwood]

To allow client applications to page through many autocomplete suggestions we should offer a search_after parameter. Calling applications would pass the last term value from the previous set of results and the returned results would be all those matching the provided prefix string param but after the supplied search_after parameter. An example might be:

Request 1

localhost:9200/myindex/_terms_enum
{
"field" : "url",
"string" : "www."
}}

Result 1

{
   "terms": [
    "www.aa.com",
    "www.abc.com",
    ...
    "www.bbc.com"
    ]

Request 2 (with new search_after param)

localhost:9200/myindex/_terms_enum
{
"field" : "url",
"string" : "www.",
"search_after" : "www.bbc.com"
}}

Result 2

{
   "terms": [
    "www.breitling.com",
    "www.bt.com",
    ...
    "www.cnn.com"
    ]

It would be a client error to pass a search_after string that does not start with the provided string prefix.

Example invalid request that would produce an error

   "search_after": "www.bbc.com",
   "string": "some_other_prefix"

Related to #71550

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[markharwood]

I'm not sure if this is viable - I coded up an implementation but was hitting UnsupportedOperationExceptions using search_after on a keyword field:

  1>               "reason" : "java.lang.UnsupportedOperationException
  1>    at org.apache.lucene.codecs.blocktree.IntersectTermsEnum.seekCeil(IntersectTermsEnum.java:566)
  1>    at org.apache.lucene.index.FilteredTermsEnum.next(FilteredTermsEnum.java:221)
  1>    at org.elasticsearch.xpack.core.termsenum.action.MultiShardTermsEnum.<init>(MultiShardTermsEnum.java:52)
  1>    at org.elasticsearch.xpack.core.termsenum.action.TransportTermsEnumAction.dataNodeOperation(TransportTermsEnumAction.java:319)
  1>    at org.elasticsearch.xpack.core.termsenum.action.TransportTermsEnumAction.lambda$asyncNodeOperation$1(TransportTermsEnumAction.java:632)
  1>    at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
  1>    at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
  1>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  1>    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
  1>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:728)
  1>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  1>    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  1>    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  1>    at java.base/java.lang.Thread.run(Thread.java:834)

The seekCeil operation is not always handled in TermsEnum implementations and the keyword field's use of automaton.getTermsEnum(terms) is an example where this doesn't work.

We only suggested offering this API because we thought it was cheap/simple to do but based on this experience I'm not sure we should offer it.

[jimczi]

The IntersectTermsEnum takes the startTerm as an argument in the constructor. That's also expose in Fields:

public TermsEnum intersect(CompiledAutomaton compiled, final BytesRef startTerm) throws IOException {

[jtibshirani]

We've had discussions on a general approach to support paging in our REST APIs: #64099. I wonder if this aligns with what we're been planning for other APIs, maybe we're standardizing on using search_after? Tagging @ywangd and @cjcenizal in case they're interested.