Add prometheus support for tensorflow-io?

[yongtang]

While in kubecon was thinking about apply machine learning (tensorflow/keras) to the field of devops. One potential area of interest is the model training/inference based on metrics data collected from prometheus which is widely used for monitoring in container ecosystem.

One challenge is that prometheus does not have a official cpp client library yet. Though it might be possible to build a shared library with golang and expose a C API to be called by tensorflow-io.

Here is a simple example of metrics collection with CoreDNS:

1. Add a Corefile and run coredns on 1053 and expose metrics/prometheus on 9153 (default):

$ cat Corefile
.:1053 {
  whoami 
  prometheus
}
$ docker run -d --net=host -v $PWD/Corefile:/Corefile coredns/coredns

2. Add prometheus.yaml and run prometheus on 9090:

$ cat prometheus.yml 
global:
scrape_interval:     15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
  - targets:
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
scrape_configs:
- job_name: 'prometheus'
  static_configs:
  - targets: ['localhost:9090']
- job_name: "coredns"
  static_configs:
  - targets: ['localhost:9153']

$ docker run -i -t --rm -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml --net=host  prom/prometheus

3. Below is the code snippet of golang code to query prometheus. Need to find out a way to build it into shared library and expose C API:

package main

import (
        "context"
        "fmt"
        "time"

        "github.com/prometheus/client_golang/api"
        "github.com/prometheus/client_golang/api/prometheus/v1"
        "github.com/prometheus/common/model"
)

func main() {
        client, _ := api.NewClient(api.Config{
                Address: "http://localhost:9090",
        })
        val, _ := v1.NewAPI(client).Query(context.Background(), "coredns_dns_request_count_total[5m]", time.Time{})
        if m, ok := val.(model.Matrix); ok && m.Len() > 0 {
                for i, v := range m[0].Values {
                        fmt.Printf("%d, %q, %s\n", i, v.Timestamp.Time(), v.Value)
                }
        }
}

Output of the above program (after some dig @127.0.0.1 -p 1053 www.google.com):

$ go run main.go 
0, "2019-05-26 18:28:56.233 +0000 UTC", 1
1, "2019-05-26 18:29:11.233 +0000 UTC", 5
2, "2019-05-26 18:29:26.233 +0000 UTC", 20
3, "2019-05-26 18:29:41.233 +0000 UTC", 20
4, "2019-05-26 18:29:56.233 +0000 UTC", 20
5, "2019-05-26 18:30:11.233 +0000 UTC", 20
6, "2019-05-26 18:30:26.233 +0000 UTC", 20

[yongtang]

Built a simple .so file with:

$ go build -o prometheus.so -buildmode=c-shared main.go

and source code:

package main

import "C"

import (
        "context"
        "fmt"
        "time"

        "github.com/prometheus/client_golang/api"
        "github.com/prometheus/client_golang/api/prometheus/v1"
        "github.com/prometheus/common/model"
)

//export Query
func Query(key []int64, val []float64) int {
        client, err := api.NewClient(api.Config{
                Address: "http://localhost:9090",
        })
        if err != nil {
                return -1
        }
        value, err := v1.NewAPI(client).Query(context.Background(), "coredns_dns_request_count_total[3h]", time.Time{})
        if err != nil {
                return -1
        }
        index := 0
        if m, ok := value.(model.Matrix); ok && m.Len() > 0 {
                for _, v := range m[0].Values {
                        key[index] = v.Timestamp.Unix()
                        val[index] = float64(v.Value)
                        //key = append(key, v.Timestamp.Unix())
                        //value = append(value, float64(v.Value))
                        index++
                        if index >= len(key) {
                                return index
                        }
                }
        }
        return index
}

func main() {
        key := make([]int64, 20, 20)
        val := make([]float64, 20, 20)
        Query(key, val)
        for i := range key {
                fmt.Printf("%d, %q, %v\n", i, model.TimeFromUnix(key[i]).Time(), val[i])
        }
}