/metrics calls on ingesters taking multiple seconds

[bboreham]

According to Jaeger traces about half are <20ms and half are 2-10 seconds.
According to Prometheus the mean is 1.5 seconds.
Traces have no detail, haven't really looked into it.

Similar: #283

[bboreham]

I added some tracing and this lead to a ReadMemStats call in prometheus/client_golang.

So I believe the root cause is (a) my ingester has a 30GB Go heap which can take more than 10 seconds to run a GC cycle and (b) ReadMemStats() is blocked for that cycle - golang/go#19812

[bboreham]

This should be fixed by prometheus/client_golang#568, when Cortex updates to that version.

[gouthamve]

This should be fixed now