Cortex return 5xx due a single ingester outage

[alanprot]

Describe the bug
Cortex can return 5xx due a single ingester failure when a tenant is being throttled (4xx). In this case, distributor can return the error from the bad ingester (5xx) even though the other 2 returned 4xx. See this.

Looking at this code seems that if we have replication factor = 2, 1 ingester down and the other 2 returning 4xx we can have for example:

4xx + 5xx + 4xx = 5xx
or
5xx + 4xx + 4xx = 4xx
etc

To Reproduce
Steps to reproduce the behavior:
I could create a unit test that reproduce the behavior:
alanprot@fd36d97

1. Start Cortex (SHA or version)
   a4bf103
2. Perform Operations(Read/Write/Others)
   Write
   Expected behavior
   Cortex should return the error respecting the quorum of the response from ingesters.
   So, if 2 ingesters return 4xx and one 5xx, cortex should return 4xx. This means that if distributor receive one 4xx and one 5xx, it needs to wait the response of the third ingester.

Environment:

• Infrastructure: [e.g., Kubernetes, bare-metal, laptop]
  Kubernetes
• Deployment tool: [e.g., helm, jsonnet]
  Helm
  Storage Engine
• Blocks
• Chunks

Additional Context

[bboreham]

Can you say a bit more about the 5xx error? The code you changed in #4388 is counting errors per-series, whereas I would expect a 5xx to be more of a whole-process error.

[alanprot]

This code counts the error per series... If i get more errors than expected, it returns the error (globally).

So lets say i have replication_set=3 and one series returns error X and Y. As the Y error was the second one (and this one is the one that breached the threshold), it will be the one returned.

So basically we return the error that breached the maxFailures of the first series that also breach it.

So, if you have one series that for some reason returned a 4xx error and after a 5xx, distributor will return a 5xx (assuming replication factor = 2).

This change is only counting the error by "Family", so if one series get a 4xx and another a 5xx, it will wait for the third ingester to decide if it need to return 5xx or 4xx.

[bboreham]

OK but how can you get a 5xx error for a single series? What's an example where this happens?

[alanprot]

Push 5XX: https://github.com/cortexproject/cortex/blob/master/pkg/ingester/ingester_v2.go#L751

Possible 4xx: https://github.com/cortexproject/cortex/blob/master/pkg/ingester/ingester_v2.go#L952

[bboreham]

That 5xx is saying the whole process is unavailable, not a single series.

If you don't need to count 5xx's per-series, that might make the fix simpler.

[alanprot]

So I guess the proposal is to return immediately on the first 4xx error (so we don't need to keep track of 5xx and 4xx separately).

The only thing to consider on this case is: we can have samples accepted for 2 ingesters and still return 4xx..

Let's consider this scenario:

• ReplicationFactor = 3
• ShardSize = 15
• Ingester 7 is running hot for a given tenant (this can happen in a normal operation as the sharding does not guarantee a 100% evenly distribution across ingesters)
• Push TimeSeries A that goes to ingester 1, 5 and 7. This request will return 4xx even though it got ingested by 2 ingesters

As discussed with @bboreham on slack in this case it may be ok to fail it back to the caller because (a) nothing changes - 2 and 3 still captured the data and (b) it's a bad state to be in - you have reduced redundancy - so it's ok to have alarms going off.

Besides that, it will behave similar to the case where 2 ingesters return 4xx and 1 return 2xx where we return 4xx to the caller and the data may still be available (but unstable) for querying: #731

What the others think?

[bboreham]

Thinking some more about this, there can be a genuine mix of outcomes across the request.
E.g. suppose you receive 100 samples for 100 series; 50 of them are errors that receive 4xx outcome, and 50 go to ingesters that have crashed so receive 5xx outcomes.
We should send a 5xx back to the caller so they can retry.

So now I'm thinking it should be a "latching" behaviour: for each series:

• a single 4xx is valid to return for that series
• fewer than [quorum] succesful results means a 5xx for that series

Across the whole request:

• all series succeeded - return 2xx
• one or more 5xx - return 5xx
• one or more 4xx and no 5xx - return 4xx

[bboreham]

It is sub-optimal that in the "mixed result" case the sender will retry everything. The remote write protocol could be enhanced to return which subset need retrying. Though I suspect this is more work than it deserves.

[alanprot]

Yeah... I'm just a little concerned about the change in behaviour here.

Nowadays if 1 ingester (for a given series) return 4xx and the other 2 returns 2xx, cortex will return 2xx.

With the proposed change, it will return 4xx, right?

[bboreham]

Nowadays if 1 ingester (for a given series) return 4xx and the other 2 returns 2xx, cortex will return 2xx.
With the proposed change, it will return 4xx, right?

Yes. For all the cases I can imagine (e.g. limit is 1000 and one ingester has 1001 series while the other two have 999), it makes no practical difference.

However I could be failing to imagine something important.

[alanprot]

@pracucci @tomwilkie Any thoughts about this?

[pracucci]

Nowadays if 1 ingester (for a given series) return 4xx and the other 2 returns 2xx, cortex will return 2xx.

Following this example, I think that if the quorum is reached if the other 2 returns 2xx and so Cortex should return 2xx.

[alanprot]

@pracucci For sure, that's the case on both solution...
but the question is, should we return 4xx on the first 4xx from ingesters or should we wait for a quorum of 4xx?

[alanprot]

Just as information those are the possible 4xx erros that can happen on the Push (ingesterV2):

storage.ErrOutOfBounds
storage.ErrOutOfOrderSample
storage.ErrDuplicateSampleForTimestamp
errMaxSeriesPerUserLimitExceeded
errMaxSeriesPerMetricLimitExceeded
errExemplarRef