RankEval doesn't take the requested number of results into consideration

[paweloque]

RankEval takes page size into consideration:

Currently RankEval only looks at the first 10 results of a query. Even if the number of results is specified, rank eval only requests 10 results and compares them with the expected results.

RankEval should consider the specified 'size' for calculations of the search quality.

[elasticmachine]

Pinging @elastic/es-search-aggs

[colings86]

@cbuescher could you take a look?

[cbuescher]

@paweloque thanks for your question, RankEval currently disregards the "size" parameter specified in queries on purpose. Instead, the maximum number of documents to be retrieved should be specified for all the current metrics in the "metric" section using the "k" parameter (named so because Precision is usually calculated for a window size k, so its named "Precicsion at k" (Prec@k).

The window size should be specified in the metric because this way we can make sure that different queries don't use different "size" values to make the results comparable. I just realized this isn't mentioned in the documentation so far, so thanks for pointing this out. I will add proper usage examples for the "k" parameter to the docs.

For now, you can use e.g.

"metric": {
    "mean_reciprocal_rank": {
      "k" : 20
    }
  }

For a result window of 20.

[paweloque]

@cbuescher Thanks for your reply and helping me! That perfectly makes sense.