Throw error when all_fields mode queries over a certain number of fields

[dakrone]

The automatic expansion of fields to query on for the query_string and simple_query_string queries has been found to have some performance issues for users with a large numbers of fields. See: elastic/kibana#12097

Should we limit the number of fields the auto-expansion is limited to, throwing an error if a query searches more than that many fields? If so, what should the limit be? @jimczi mentioned 100 fields in the original issue as perhaps a good starting point.

[dakrone]

We discussed this, and decided that we should add the same limit that BooleanQuery has to the DisjuntionMaxQuery - that is, a limit of 1024 clauses.

Additionally, we should document that the all_fields mode can be expensive in the case where there are a lot of fields, and that a user should search a subset of fields or use copy_to to create a custom field containing the contents of other fields.

[trevan]

@dakrone, Could there also be a way to restrict which fields are included in an "all_fields" query? I have over 1400 fields in an index and I've been planning on removing the _all field to reduce memory usage. But this restriction will prevent easily searching without specifying a set of field names. I'd be fine with determining which fields shouldn't be used in "all_fields" to reduce the number below <1024 but I don't want to use copy_to since that is basically the same thing as using the _all field and the memory usage won't be reduced.

[dakrone]

Could there also be a way to restrict which fields are included in an "all_fields" query?

You could use the multi_match query and specify the fields manually in the request (so like "fields": ["foo.*", "bar," "baz", "eggplant"]) to restrict them, just make sure you set "lenient": true also. Either that, or you could use copy_to to create a custom field that contains only the fields you want it to contain (a custom _all field)

[mr-mos]

@trevan @dakrone I would vote for restricting fields in the "all_query". A multi_match or other queries would expect, that clients know which fields are available and should be searched by default. In many cases this is a configuration on the index and shouldn't be known by the client.
Please check also our current discussion: https://discuss.elastic.co/t/option-for-excluding-fields-in-all-fields-mode-query-for-es-6-0/93218/4
Should I open an issue for this?

[dakrone]

Should I open an issue for this?

@mr-mos sure, please do open an issue so we can discuss it!

[mr-mos]

@dakrone Opened the issue #25946
(Wasn't able to set any lables?)