Additional labels for alertmanager target types in log alert spec

[nitisht]

We could also add some additional labels to add more context from where the alert is coming from.

parseable/server/src/alerts/target.rs

Line 261 in c15cdd1

impl CallableTarget for AlertManager {

Suggested additional labels:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

__address__ = config.parseable.address
__scheme__ = config.parseable.get_scheme()
__alerts_path__ = /api/v1/logstream/{{stream_name}}/alert

__meta_parseable_deployment_id = StorageMetadata.deployment_id
__meta_parseable_version = StorageMetadata.version
__meta_parseable_mode = StorageMetadata.mode
__meta_parseable_alert_rule_type = "column"
__meta_parseable_alert_rule_config_column = "status"
__meta_parseable_alert_rule_config_operator = "notEqualTo"
__meta_parseable_alert_rule_config_repeats = 2
__meta_parseable_alert_rule_config_value = 500

Originally posted by @syepes in #306 (comment)

[nitisht]

@syepes how about clubbing the whole alert reason into a single label, like so?

__meta_parseable_alert_reason = "status not equal to 500, 2 times"

[syepes]

@syepes how about clubbing the whole alert reason into a single label, like so?

__meta_parseable_alert_reason = "status not equal to 500, 2 times"

That also would work but also having the __meta_parseable_alert_rule_type to know the type

[nitisht]

Thanks @syepes makes sense. We looked at templating in alertmanager and now we agree on your initial proposal of separate labels for each rule field.

This will allow users to template their rules and notifications exactly how they need it. Much more powerful this way.

[syepes]

No problem, that was the initial idea to not have to re-parse the labels.

[syepes]

@nitisht
This is now working pretty good, the only missing label would be the instance = config.parseable.address + port, so we can go back to the parseable instance that generated this alert:

#316