Allow configuring retention settings for the automatic indexes (e.g. otel-logs-v0_X)

[to266]

Is your feature request related to a problem? Please describe.
We use quickwit to ingest unfiltered logs and traces from our clusters. We exclusively use the opentelemetry to do both, and due to the total amount of data, we definitely want some kind of retention policies for everything

Describe the solution you'd like
A config in the official helm chart, which would allow configuring such settings for the automatically created indexes

Describe alternatives you've considered
Right now what I do every time a new index is automatically created, is basically use the REST API to first get the full configs of the indexes (and I'm generally mostly OK with whatever is the default), and then POST /api/v1/indexes?overwrite=true with basically the same config + my retention settings.

That's both cumbersome / inconvenient, as well as deletes all the data that was collected up to that point, since it forces index recreation, as no PATCH (with a merge) are available.

There's also no possibility to actually completely hand-tailor a custom index, as far as I can tell, since OTEL data only lands in the two automatic indexes

[guilload]

Which metastore are you using? Until we provide a better experience for updating index configs, you can edit them directly in the metastore files for the file-backed implementation or via SQL for the Posgres metastore. Then, you can restart the janitor, and you should be good to go.

[fmassot]

Adding a default retention on the OTEL indexes is a good idea, I would go for 30 days as the default, @guilload what do you think?

We will soon allow updates on index config so this should help.

In the meantime, you have several possible solutions:

• it's possible to use custom OTEL indexes with the gRPC OTEL API, by specifying a header (see doc example), we added this possibility in the 0.7 version.
• It's possible to use also the http-proto OTEL API too, here you can specify the index by choosing the right endpoint {your-index}/otlp/v1/logs or {your-index}/otlp/v1/traces

[to266]

Hm, I'll have a look at the solutions you suggest @fmassot. I admit the last time I attempted, it was still with the v0.6.x.

That being said, we're not really unhappy with the default indexes, with the one exception of retention for sufficiently busy clusters.

Regarding modifying metastore (and we're using postgres @guilload) - yes, I guess that's another possibility, maybe a tiny bit easier (and more brittle, though does not delete the data) than using the official API. But the whole point is that I don't really want to do it manually after the pods are started. What I really want, is configure it in my GitOps repo, and whatever mechanism further applies the helm release, it should be sufficient, with no manual intervention. At least in the best-case scenario 😄

[fulmicoton]

30 days sounds good.

[guilload]

Ok, understood, @to266. Thanks for your detailled answer. We can ship a few things in 0.8 to make your life easier:

• We will support updating some parameters of index configs via the REST API
• We will support templates for OTEL indexes, which you can define and customize in our Helm chart.