Obtain tenant ID from Prometheus timeseries label

[blind-oracle]

We're using several HA pairs of Prometheus and want to be able to ingest to different tenants from a single HA pair based on labels in a trusted environment. We need this to separate teams/application groups.

Currently out of box this is not supported and I wrote a small proxy that sits between Prometheus & Cortex and does the job (https://github.com/blind-oracle/cortex-tenant). It looks for a predefined label and uses it as a tenant ID.

Do you think it's worth to implement something like this in Cortex directly somewhere in the Distributor?

[roidelapluie]

There is a auth gateway in the roadmap

https://cortexmetrics.io/docs/roadmap/

[blind-oracle]

@roidelapluie

Yeah, but from proposed functionality it doesn't look like that it would allow multiple tenants per single Prometheus...
But anyway, if such entity is planned then probably this feature should go there.

[pracucci]

There are thoughts about adding relabeling support during ingestion (in distributors). This use case could be a good candidate for a custom relabelling rule (where you can change the tenant ID). @gouthamve thoughts?

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[blind-oracle]

Any chance on this?

[pracucci]

Any chance on this?

No progress on this yet. I think doing it at a proxy level is still the only (and probably) best way to do it. In a previous message I suggested whether we could use relabelling support but, after more thoughts, I'm not sure it's easy to fit into the distributor.

[roidelapluie]

Any chance on this?

No progress on this yet. I think doing it at a proxy level is still the only (and probably) best way to do it. In a previous message I suggested whether we could use relabelling support but, after more thoughts, I'm not sure it's easy to fit into the distributor.

I tried that but many questions rise: which limits to apply, should the metrics be fed twice in the distributor ? (and how to prevent "loops" then), etc..

[pracucci]

I tried that but many questions rise: which limits to apply, should the metrics be fed twice in the distributor ? (and how to prevent "loops" then), etc..

I see the pain and I think it doesn't really fit. The whole Cortex has been designed to receive the tenant in the input request, so the distributor (like other services) have been designed this way. I think doing it at a proxy level is still the right (and best) way to do.

[blind-oracle]

Ok, thanks guys, then I guess let's leave it as is then for now and use the proxy 👍

[blind-oracle]

@pracucci btw maybe it's worth to add a link to the proxy in docs somewhere, probably it would be useful to somebody until the Auth gateway is released. Maybe even after that, depending on the use case.

[pracucci]

@pracucci btw maybe it's worth to add a link to the proxy in docs somewhere, probably it would be useful to somebody until the Auth gateway is released. Maybe even after that, depending on the use case.

@blind-oracle Not sure I follow you. Using a custom proxy (to be built based on your specific use case) was a suggestion, but we haven't built that proxy, so I'm not sure what we should link in the doc.