Unexpected behaviour if using nested variables in grafana templates with block storage

[kubicgruenfeld]

We are using the experimental block storage and have an issue with nested variables in Grafana.

If using template variables in Grafana like label_values(node_load1{hardwarenode='$hardwarenode'}, instance) and try to query data which is not present in the ingesters anymore, there are no labels found.

If templates are using no nested variables like label_values(node_load1, instance), there is no problem, even querying old data as long as you use Refresh: 'On Dashboard Load' in Grafana.

There are some screenshots attached to show the behaviour.

Querying last 7 days with nested variables is working:

Querying a timerange not present anymore in ingesters there are no labels found for this timerange:

It looks like the labels are tried to be retrieved from ingesters for the selected timerange. Since labels are only queried from ingesters, there are no labels found for this timerange and therefore empty. I am not sure if this is expected behaviour or an issue.

[pracucci]

Thanks for reporting it. Looks an issue. I'm investigating it.