Add support for Prometheus 2.0 rule format

[juliusv]

The rule format to use is now set binary-wide via the
-ruler.rule-format-version and configs.rule-format-version
flags, which still default to the Prometheus 1.x rule format.

There's some trickiness here regarding what data type to return from parsing,
regarding the ability to track alert states, and not being able to create final
rule groups yet. That's laid out in the comment above RulesConfig.Parse().

Fixes #622

[juliusv]

This is based on the vendoring updates in #688.

@jml This is basically ready and working, but marked [WIP] only because:

• I need to make slight readability improvements to error reporting to the user.
• I need to check whether we want to change the evaluation metrics a bit, due to the fact that we're now executing multiple groups per user, with multiple rules each.
• We can't just deploy this without having a conversion plan for existing rule files in place.

[jml]

Great, thanks.

I'm stretched a bit thin, so I'll wait until you've addressed the first two points and try to find someone else to look into conversion plan. If I can't find someone else, will do it myself.

[juliusv]

@jml I rebased ontop of the latest master and added a couple more fixups and improvements.

The Prometheus rules YAML parser can return multiple errors if it finds multiple problems, but I wasn't sure how to best present those to the user, so I simply opted to just always show the first one (so then the user can still iteratively fix their errors).

In terms of metrics, we are still measuring the durations of each rule group (as before), except that previously all rules for a user were dumped into one large group, whereas now a user can specify groupings themselves in the new YAML rules config. So you will see more, but smaller groups, and accordingly faster per-group evaluation durations. The metric that tracks the overall latency of a scheduler work item completion is still the same, except for a rename / help text update to reflect that it's not about one rule group, but a whole set of them for a given config.

I think this should be ready from a code perspective now, but I'm keeping the [WIP] so that nobody accidentally merges it before we have a transition plan. A transition plan should include converting all existing user configs to the new format, updating example/default configs, Cortex documentation around configs, and maybe notifying users of the change.

Do we also want to rename the current "prometheus-1518408565633.rules"-style names to end with ".yml"?

[jml]

Thanks Julius! No movement on a transition plan. I haven't found anyone w/ spare cycles to think about it. Will keep pinging. Renaming seems sensible.

…

On Mon, 12 Feb 2018 at 05:40 Julius Volz ***@***.***> wrote: @jml <https://github.com/jml> I rebased ontop of the latest master and added a couple more fixups and improvements. The Prometheus rules YAML parser can return multiple errors if it finds multiple problems, but I wasn't sure how to best present those to the user, so I simply opted to just always show the first one (so then the user can still iteratively fix their errors). In terms of metrics, we are still measuring the durations of each rule group (as before), except that previously all rules for a user were dumped into one large group, whereas now a user can specify groupings themselves in the new YAML rules config. So you will see more, but smaller groups, and accordingly faster per-group evaluation durations. The metric that tracks the overall latency of a scheduler work item completion is still the same, except for a rename / help text update to reflect that it's not about one rule group, but a whole set of them for a given config. I think this should be ready from a code perspective now, but I'm keeping the [WIP] so that nobody accidentally merges it before we have a transition plan. A transition plan should include converting all existing user configs to the new format, updating example/default configs, Cortex documentation around configs, and maybe notifying users of the change. Do we also want to rename the current "prometheus-1518408565633.rules"-style names to end with ".yml"? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#689 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHq6qZ-bI1xfTXHxGjBaN7GcT10cZ7Yks5tT87BgaJpZM4R6zUO> .

[juliusv]

@jml I completely re-did this PR ontop of #719, which touched all the same code places.

I also added flag-based binary-wide support for setting the rule format, with it still defaulting to v1. Ideally this should be deployable without breaking anything unless you explicitly set flags to indicate a v2 rule format.

[bboreham]

Just nits really

[bboreham]

Why is this here? Could we have something more descriptive than "VERSION"?

[juliusv]

Whoops, this was just local dev debugging output. Removed.

[bboreham]

Could we have the indentation the same between v1 and v2 here? For consistency, and I think this would also improve the diffs.

[juliusv]

Done, I outdented this to what it used to be.

[bboreham]

Maybe factor this switch out to a function in config, so we don't have to edit this file when v3 is added?
(And it would help keep down the complexity of this function)

[juliusv]

Great idea, I factored that into a Parse() method in config, which also got rid of those switches in other places.

[bboreham]

hmm

[juliusv]

Panics on programming errors are pretty normal/ok, no? Best way to actually catch them quickly...

[bboreham]

@juliusv can you check your dep command please - this PR seems to bring in a bunch of *_test.go files which were removed in #705. Maybe you have an older version?

[juliusv]

@bboreham As discussed on Slack, I rebased this PR ontop of latest master, ran dep ensure (with newest dep version) again, and squashed the extra vendoring changes into the original vendor update commit.