Give _uid doc values

[jpountz]

We already use fielddata on the _uid field today in order to implement random sorting. However, given that doc values are disabled on _uid, this will use an insane amount of memory in order to load information in memory given that this field only has unique values.

Having better fielddata for _uid would also be useful in order to have more consistent sort order when paginating or hitting different replicas: we could always add a tie-break on the value of the _uid field.

I think we have several options:

• Option 1: Add SORTED doc values to _uid
• Option 2: Add BINARY doc values to _uid
• Option 3: Add SORTED doc values to _type and _id
• Option 4: Add SORTED doc values to _type and BINARY to _id

Option 2 would probably be wasteful in terms of disk space given that we don't have good compression available for binary doc values (and it's hard to implement given that the values can store pretty much anything).

Options 3 and 4 have the benefit of not having to duplicate information if we also want to have doc values on _type and _id: we could even build a BINARY fielddata view for _uid.

Then the other question is whether we should rather use sorted or binary doc values, the former being better for sorting (useful for the consistent sorting use-case) and the latter being better for value lookups (useful for random sorting).

[pickypg]

My vote would be for number three or four:

Option 3: Add SORTED doc values to _type and _id
Option 4: Add SORTED doc values to _type and BINARY to _id

With #14783 we already enable doc values for _type, so it makes sense to individually call out the _id as well. This also allows changes to happen to _type without necessarily breaking _id.

In my experience, most users do not use random sorting, but sorting on _id is not very common either. With only sorting in mind, I would expect to see _id used for non-random sorting a lot more than for random sorting. However, for other use cases, such as referencing the _id in other scenarios that use fielddata (e.g., rarely, but sometimes in aggregations, as well as scripts), it may tip it in favor of being binary.

[rmuir]

There is a huge difference between _type and _id when it comes to the expense of docvalues.

_type is a lowish cardinality field. This means if you have only 2 unique values foo and bar, lucene will deduplicate this and write 1 bit per document. If you only have 1 unique type (also common), we will write 0 bits per document, each segment just has in the metadata "all docs have value foo: it costs nothing). So for 10M documents with 2 types, docvalues for this field costs a little over a megabyte.

On the other hand unique ids are high cardinality by definition: deduplication does nothing. Either choice is extremely costly in comparison. Lets consider 10M documents with ids of length in bytes 16 each and make some guesses:

• BINARY might you ~ 160MB for the bytes (10M * 16). That is how binary works: its just a straightforward encoding of what you gave it. If the IDs do not have a fixed length, but are instead variable length, then there are additional costs.
• SORTED might cost you ~160MB for the bytes (10M * 16) and additional 30MB (10M * 24bpv) for ordinals. The bytes are prefix compressed in this case, because access by ordinal is more important, but for randomish ids this compression will probably not be very efficient. Access to the bytes is also slower, that is the downside of prefix compression (which likely does not help).

I just want to make it clear this is apples and oranges. The fact we turned on docvalues for type is irrelevant when it comes to unique ids. We need very strong use cases and features IMO if we are going to incur this cost.

[pickypg]

Very good info @rmuir, as usual. It makes me think that _id supporting doc values should exist (particularly in light of #15155), but it should be opt-in.