Fetching metrics from ingester is really slow

[tomwilkie]

[tomwilkie]

We're seeing this again, super annoying. Its on an ingester with a lot of series in memory, which suggests it could be the counting of the number of series: https://github.com/weaveworks/cortex/blob/master/pkg/ingester/user_state.go#L101

Or it could be the minimum of 3 locks that we have to grab to fetch metrics from the ingesters; the block profiler might help us find out.