Support metric_relabel_configs in distributor

[roidelapluie]

Signed-off-by: Julien Pivotto [email protected]

What this PR does:
Support metric_relabel_configs in distributor

Which issue(s) this PR fixes:
Fixes #1507

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[csmarchbanks]

🥳, I am excited for this!

Do you think it is worthwhile to allow this to be specified per user rather than just globally? I imagine a common use case will be to stop very high cardinality or high churn data from a specific tenant without disrupting data for all tenants.

[drequena]

configs instead of confis (just a simples typo) ;)

[pstibrany]

Can we move the configuration into Overrides? Then it can be per-user, and reloaded when changed on disk, without restarting Cortex. WDYT?

[pstibrany]

We should note that this is global, for ALL tenants.

[roidelapluie]

I was already thinking about #3318 (comment) when I set this up. WDYT? Should we have both?

[pstibrany]

I was already thinking about #3318 (comment) when I set this up. WDYT? Should we have both?

If by "both" you refer to global and per-tenant config, then Overrides / Limits already support that, and yes, I think it makes sense to have both. For example in cloud offering, we wouldn't use global one, but could use per-tenant config in specific cases. On the other hand, some people running Cortex internally may only need global one.

[bboreham]

Thanks! Any idea what the CPU/memory overhead is?

[roidelapluie]

That depends on the relabeling rules..

[roidelapluie]

I have moved it to limits

[jtlisi]

LGTM!

[jtlisi]

question: Is there any instrumentation or debug logging that could be added that would be helpful for operators/users when working with this feature?

[roidelapluie]

IMHO the metrics distributor_received_samples_total and distributor_samples_in_total cover that already.

[jtlisi]

Suggested change

	* [FEATURE] Support `metric_relabel_configs` in distributor. #3329
	* [FEATURE] Added the support for applying a Prometheus metrics relabel config on series received by the distributor. The `metric_relabel_configs` field was added to the limits config. #3329

[roidelapluie]

Thanks, I updated my comment

[pstibrany]

Thank you, this looks very good! Would it be possible to add small test showing parsing of relabeling YAML rules into per-user limits?

[roidelapluie]

Thank you, this looks very good! Would it be possible to add small test showing parsing of relabeling YAML rules into per-user limits?

Do we have tests like this already?

[pstibrany]

Thank you, this looks very good! Would it be possible to add small test showing parsing of relabeling YAML rules into per-user limits?

Do we have tests like this already?

Until now structures we've put into overrides YAML file were pretty basic. I can only see TestLimitsLoadingFromYaml. Something similar in size but with relabel config to make sure the wiring is all correct would be good to have. WDYT?

[roidelapluie]

Done :)

[pracucci]

Could we mark it experimental in docs/configuration/v1-guarantees.md, please?

[roidelapluie]

done

[pracucci]

Thanks for working on this and sorry for the super late review (I would like days of 48h 😢 ). Looks flawless to me. I just left a nit, definitely not a blocker.

[pstibrany]

Thanks a lot Julien!

[bboreham]

I just used this feature in production to remove a high-cardinality metric; worked super well!

Even better if we had a metric to track how many samples got dropped this way - right now they disappear from cortex_distributor_received_samples_total and don't appear in cortex_discarded_samples_total.