Investigate other ranking evaluation metrics

[cbuescher]

From an old dicussion in our forums I just learned about another interesting looking ranking evaluation metric used in some TREC competitions called "bpref" that is advertised to work well with incomplete data.

I'm opening this issue to do some more investigation into this and other evaluation metrics that we haven't considered yet.

Regarding bpref its atm. unclear to me:

• how widely used it is
• in which use cases it might perform better than the metric we currently offer
• if we can implement it with our current API that is based in msearch or if we would need to change something to make it work

[elasticmachine]

Pinging @elastic/es-search-aggs

[rpedela]

I would love to see expected reciprocal rank (ERR) added.

[cbuescher]

@rpedela great suggestion, I will look into this as well and how it fits into the current design of the API. Do you already use ERR? If so, for which kind of use case and how does it compare to other metrics (like e.g. nDCG) in your experience?

[rpedela]

Doug Turnbull from Open Source Connections does a great job answering your question in this talk starting at 21:18.

[rpedela]

One more data point. Ranklib is the de facto learning to rank library and ERR is the default optimization metric used for training.

[cbuescher]

@rpedela I started looking into ERR and found it to be a great additional metric. I've opened a PR at #31891, maybe you'd like to comment if you are familiar with the calculation of this metric and want to check if my understanding of the algorithm is correct. In particular I was wondering about the handling of ungraded search results. The paper assumes complete labels but this is unrealistic in a real-world scenario. For now I opted for an optional, user-supplied "unknown_doc_rating" parameter that gets substituted for search results without a relevance judgment (it could simply be 0 for most cases). If this parameter is not present, unrated documents are just skipped over in the metric calculation. Not sure if that is common practice but would like to hear thought or get pointers on this.