[FEATURE][ML] Fetch from source when fields are more then docvalue limit

[dimitris-athanasiou]

There is an index level setting called index.max_docvalue_fields_search which limits the number of doc_value fields that can be returned from a search. This commit changes behaviour of the extractor so that it's now reading the value of that setting and if there are more fields we switch to fetching the fields from _source.

[elasticmachine]

Pinging @elastic/ml-core

[benwtrent]

I think this adds bugs for TimeField and GeoPointField extractors.

[benwtrent]

Shouldn't this be extractedFields.getDocValueFields().size()?

[dimitris-athanasiou]

A few lines above we filter only doc_value fields, so it makes no difference. But I see how it'd be clearer if I made this change. I shall!

[benwtrent]

It seems to me that we only really need to transform as many as necessary to drop below the docValueFieldsLimit, and then only those that that supportsFromSource().

[dimitris-athanasiou]

we only really need to transform as many as necessary to drop below the docValueFieldsLimit

As long as we have to touch the source, we might as well fetch them all from there. Performance-wise it's going to be better.

and then only those that that supportsFromSource()

But this is a very good poing. In case we have fields that don't support from-source, we should insist taking them from doc_values. Then we'll also need a check to see if we're still over the limit.

[benwtrent]

So, we are assuming that anything that is FromFields that has ExtractionMethod.DOC_VALUE also supports extraction via _source? This seems OK to me for the most part, except for GeoPointField, which may need to override supportsFromSource() to always return false.

I think we may want to do something similar with TimeField

[dimitris-athanasiou]

Good catch! Will adjust.

[benwtrent]

🏑 🌱