Autocomplete suggestion no longer removes duplicate entries as in ES 2.3 !!

[seme1]

I relied on the autocomplete suggester in 2.3 to remove duplicate entries and provide unique suggestions of words/phrases. After upgrading to ES 5, I realized that the suggester is now document-oriented. This changes the logic of the how it can be used with already-built systems. Also, it seems to me that a feature was removed from ES 2.3. In other words, this is not the upgrade I was expecting !!

It also seems that I'm not the only one suffering from this !!
#21676
http://stackoverflow.com/questions/41744712/word-oriented-completion-suggester-elasticsearch-5-x

[seme1]

The "Duplicate filtering" feature outlined in the following article is no more in ES 5
http://rea.tech/implementing-autosuggest-in-elasticsearch/

I'll stick with ES 2.3 till a resolution for this missing feature is found.

[clintongormley]

@seme1 this change was a requested feature. The design of the new completion suggester is significantly different to what existed before - a suggestion maps to a single document now. This isn't going to change.

I'm afraid your only option is to remain on 2.3 until you've had time to rework your application.

[seme1]

How to "rework my application" ? I am relying on elasticsearch to give users suggestions on what terms/phrases to type when they search. I don't want the auto complete suggestion to give results pointing to individual documents, but rather to the most common/important terms/phrases.

[clintongormley]

I don't want the auto complete suggestion to give results pointing to individual documents, but rather to the most common/important terms/phrases.

This was never the intention of the completion suggester - it has never favoured the most common words etc. It is a prefix suggester: it looks at the prefix you've typed in and finds completions from a finite list of strings with exactly that prefix.

If you want to use the suggester to suggest search terms which may be duplicated across documents, then you need to index those search terms in a separate index and handle the deduplication yourself.

[uschindler]

Sorry this is still a desaster! Is there no way to add deduplication back? This prevents some customers (and myself) to update to Elasticsearch 5. Collecting all suggest items (deduplicated) into a separate index is not going to work, as it again prevents easy updates.

[uschindler]

If anybody is interested: I wrote a plugin with a hack that restores the old Completion Suggester. The trick was to add a new field type "legacy_completion" that behaves identical to the 2.x version of this suggester. It is a bit of hack, but works. I will post a GIST soon to show what it does - it is included into my own plugin so I have to extract this first.

Nevertheless: As all this works fine, why not keep the old completion suggester alive for people that are not interested in "document" suggestions, but want Google-like term autocompletion, but still want to have all in one index. The old suggester is perfect for that and the fact that deletions are not taken into account is not a problem at all (for this use case). In addition the deduplication works fine and is fast enough! Payloads are not required for this use case.

I'd suggest to add a field type like my plugin "legacy_completion", but without hacks. I'd suggest to remove payloads for this use-case, too.

[mikemccand]

Indeed the new suggester (called the document suggester in Lucene) is document based and does not have any ability to remove dups today.

There was some discussion early on about duplicates: #22912 (comment) but I don't think it led to any duplicate removal being added.

@areek can you confirm?

I suppose we (or users) could add a while loop to query for suggestions, and then iterate (with a larger N) if too many of the top N on the first try were duplicates.

[uschindler]

I'd be happy to keep the old suggester available with own field type, like my plugin is doing. I will post the plugin code tomorrow. It is quite simple, but a hack that breaks easily if internals change or the old suggester code is completely removed in ES 6. That is what I would like to prevent.

[mikemccand]

I think we can also add duplicate handling to this suggester as an option in Lucene? I'll open a Lucene issue to explore it; I think it may not be so difficult.

[mikemccand]

I think we can also add duplicate handling to this suggester as an option in Lucene? I'll open a Lucene issue to explore it; I think it may not be so difficult.

OK, I opened https://issues.apache.org/jira/browse/LUCENE-7686 to add optional deduplication to the document suggester, and iterated to a working patch I think.

[uschindler]

Hi Mike,
thanks for opening the Lucene issue! It looks good to me. But nevertheless a Suggester that is document based is not always the best idea. Your patch on the Lucene issue already mentions it: If you have lots of duplicates, this slows down and the idea behind a suggester is broken (as its slow, possibly horrible slow). Let me explain the 2 different types of Suggester "use cases":

• The new suggester is document based, that is fine if you really want to suggest documents (and also want to filter deleted documents). Basically this suggester just executes a query and returns TopDocs (not really Topdocs, because it uses another scoring, but basically it is the same). This type of suggester works fine if you index for suggestions is unique, e.g., the document title (that should be almost unique) and suggest those in the drop down. If users click on those items, they are directly directed to the document. One example of this is the search engine on Elastic's home page. The risk of duplicates is small there.

• The old suggester was just "dictionary based" (a variant of the term dictionary that has payloads and some weights). This suggester does not suggest documents, it just suggests terms/phrases you could enter into the search field and execute them. The same like Google is doing - and this is what I would need (and the user who opened the issue). Generally you can do the same also with the new suggester, but you must take care of using a separate, deduplicated index and index the suggester phrases from there. But this makes maintenance hard! If you have structured data and you know that some fields in you documents are useful as auto-suggestion (e.g, names of authors, journal names,... anything that could also work as a facet), then you are done already. This helps users to enter such terms (which are real suggestions, documents are no suggestions, they are already results of search). The problem with stuff like author names is that they may appear in thousands of documents. If you execute the search afterwards you get thousands of documents. This is how you would use a suggester like you see it on Google: just present phrases to search for which may return many documents. For this use case a Document-based suggester as the new one here is not scalable. To make it work as it should, user would need to create 2nd index, then execute an aggregation on the primary index on a field used for suggestions and migrate the buckets of the terms aggregation as suggestions. The backside: This makes adoption hard, as you have to run this in regular intervals and reindex the suggester index. For users with a high frequency of coming/going documents thsi is not gonna to work. The old completion suggester had the deduplication "automatically", because it just worked on the in-memory terms dictionary, but never iterated over documents. The backside: deleted stuff does not disappear. But for this type of suggestion this is not necessarily to happen in a timely manner. If you get a suggestion which is no longer in the index as the "whole phrase", it is still likely return results (for multi-word suggestions). So this was never a problem for the old suggester. I had never any problems with deletions not showing!

My suggestion would be to use the new suggester's dictionary, but just allow it to be run without return document suggestions: Just find the terms/phrases from dictionary that match and return them as suggestions. The score could be document frequency. That would help both worlds. The alternative would be to keep the old suggester alive as alternative (as said before). To me this would work like the still missing Solr "terms component" in ES.