_id-less indices

[jpountz]

Various experiments over time have highlighted that the _id field, and to some extent the _seq_no field, use non-negligible disk space. When indexing tiny documents, these two fields combined can end up using more than 50% of the index size. Are there conditions in which we could enable users to drop the _id field in order to save resources?

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Distributed)

[jpountz]

We discussed two special kinds of indices that could work without ids:

• Read-only indices. These would work by creating a regular index, adding data to it and at some point turning it read-only. This operation would be non reversible and could allow to get rid of the _id, _seq_no and _primary_term fields, since those are not needed for searching. After an index is turned read-only, indexing, GET, delete and update operations would be rejected.
• Append-only indices. These indices would need to be created as append-only from the beginning and would only allow append index requests (POST index/_doc). GET, delete and update operations would be rejected. It would work internally by dropping the _id, _seq_no and _primary_term fields as part of the merge process on all segments whose sequence numbers are part of no retention lease.

Read-only indices would be much simpler to implement and would fit nicely into index lifecycle management. Its main drawback is that savings are delayed to a later time, so this might not address the perception that Elasticsearch uses a lot of disk space, and Elasticsearch would still perform poorly disk-usage-wise in benchmarks unless turned into a read-only index at the end of the benchmark. Append-only indices would address these points but also bring more complexity in order to avoid duplicates because of client-side retries, or because of retries on the coordinating node.

There is potential for better compression of these fields, but reducing by 20% would already be huge and would only reduce the overall index size by 10% assuming that _id/_seq_no take 50% of the index size, which is why we are rather looking at removing these fields completely. We will probably want to look into some compression improvements, but this is not the focus of this issue.

Since the relative overhead of _id/_seq_no is mostly noticeable when documents are very small, a point was made that we could also look into providing better tooling to help e.g. merge several measurements into a single document in order to have fewer, larger documents.

Next steps:

• Get a better sense of the complexity of append-only indices and whether their benefits are worth this complexity.
• Better understand when users create small documents and whether better tooling could help create fewer larger documents.

[martijnvg]

Dropping the _id field in the context of data streams makes perhaps more sense than just on regular indices. Currently a data stream only accepts append-only writes, but updates/deletes are allowed directly via the backing indices. We can think about introducing new data stream specializations then we can better enforce id-less / append-only writes.

I think there is a place for both kinds of _id removal in the context of data streams. In case that updating (or client retries) should be allowed for a long period of time then perhaps a backing index should drop the _id when it is effectively read only (we could then drop _id, _seq_no and _primary_term as part of the force merging performed by ilm). In the case that client retries aren't that important then the _id field can be dropped as soon as the document's seqno is past the minimum retained seqno (so _id is available for replica replication and ccr). Even in the latter case, client retries could still be possible if the previous write's seqno is greater than the minimum retained seqno (the window of operations could be kept large if that is required in certain setups).

Both kinds of implementations would be centered around a merge policy that drops the _id and new data stream semantics, so I think having both options available shouldn't add that much work.

[jpountz]

Some data points regarding how much we can expect to save with this change:

• On an index containing nginx.access data, _id and _seq_no were responsible for respectively 4.5% and 2.5% of disk usage, ie. a total of 7% of disk usage.
• On an index containing metrics pulled from Prometheus, _id and _seq_no were responsible for respectively 8% and 5% of disk usage, ie. a total of 13%.
• On an index containing Elasticsearch logs, _id and _seq_no were responsible for respectively 25% and 1% of disk usage, ie. a total of 26%.

[tsg]

On an index containing Elasticsearch logs, _id and _seq_no were responsible for respectively 25% and 1% of disk usage, ie. a total of 26%.

I'm curious about this one because the _id percentage seems very large. Are Elasticsearch logs ingested with the Elasticsearch module in Beats or directly ingested with just the message field or similar? Is there a place where I can see the test data?

[jpountz]

@tsg The data can't be made public but I'll share more details privately with you.

[jpountz]

One aspect that I had not considered until now is that dropping _id upon merge might help with the indexing rate, as follow-up merges wouldn't have to care about the _id field.

[timroes]

I am trying to understand the full-impact of that change, and there is one thing not entirely clear to me yet: If we'd have _id less datastreams, what would be the way to uniquely identify a document in them? Or wouldn't there be any way in those data streams anymore?

We rely in Kibana in a lot of places on uniquely identifying objects by _id (and index) at the moment, though we could exchange that by another unique identifier, but would have problems, if there wouldn't be any unique identifier available for documents.

A non exhaustive list of features currently using _id (and would require some unique identification of documents):

• The Maps application can show a layer of individual documents on a map, when opening a tooltip of one of those documents we load the other (non Geo fields) we want to show about that document via a separate GET request to get them.
• In Discover you can "View a single document". We link to something like /view/<index>/<id> and when accessing that view, use an id query to load that document to show. Similar we load the document for "Viewing surrounding documents".
• In Discover you can look at documents in a flyout, not just the configured columns. We are/will use id queries there to load the document to show.
• We require in our frameworks often to specify a unique identifier when rendering a list of rows (i.e. documents), so the framework can keep track of those rows, e.g. when scrolling. We're using the document _ids (in conjunction with indexes) to keep track of them.

[weltenwort]

What @timroes mentioned for discover applies similarly to the Logs UI. We use the _id to disambiguate log entries when fetching them individually.

We rely in Kibana in a lot of places on uniquely identifying objects by _id (and index) at the moment, though we could exchange that by another unique identifier, [...]

Even if we had a different way to uniquely fetch specific docs it would ideally have to be applicable across all types of indices (datastream-backing or not). Otherwise separate code paths could represent an increased maintenance burden for many such single-document use-cases.