Realtime Monitoring System Prototype in Golang, Kafka, TimescaleDB

Read the blog for more implementation details: [TBA]

Demo

output.mov

Tech Stack

• Golang: For all services.
• Kafka: Message broker for real-time data streaming.
• TimescaleDB: Time-series database for storing metrics.
• Docker & Docker Compose: Containerization and orchestration.
• SMTP (Gmail): Email notifications.
• REST API: Service communication.

High Level Design

Run this project

• Create .env file in the root folder.

      ~/ cd golang-monitoring-system
      ~/ touch .env

• Update the .env variables, for testing email go to Google Account App Password and create one.

      # Kafka
      KAFKA_BROKER_ID=1
      KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
  
      # TimescaleDB
      POSTGRES_USER=<user>
      POSTGRES_PASSWORD=<pwd>
      POSTGRES_HOST=timescaledb
      POSTGRES_PORT=5432
      POSTGRES_DB=metricsdb
  
      # App configs
      KAFKA_ADDR=kafka:29092
      KAFKA_TOPIC=golang.monitoring.system.metrics
      CONSUMER_GROUP=ingest.consumer.group
  
      # Notification configs
      QUERY_SERVICE_ENDPOINT=http://query_service:8083/qs
      SMTP_HOST=smtp.gmail.com
      SMTP_PORT=587
      SMTP_EMAIL=<your_email>
      SMTP_PASS=<app_password>

• Use Docker & Docker Compose for running the project.

      ~/ docker compose up --build -d
      ~/ docker compose down

API Reference

• App Service APIs:

      curl -X GET http://localhost:8080/metrics/cpu
      curl -X GET http://localhost:8080/metrics/memory
      curl -X GET http://localhost:8080/load/cpu\?m\=2
      curl -X GET http://localhost:8080/load/memory\?m\=1

• Metrics Collector APIs:

      curl -i -X GET http://localhost:8081/ping
      curl -X POST http://localhost:8081/register -d '{"endpoint":"http://app_service:8080/metrics/cpu", "interval": 5, "metric_name": "cpu", "service_name": "app_service"}'
      curl -X DELETE http://localhost:8081/stop?s="app_service"&m="cpu"
  
      docker exec -it <kakfa_container_id> kafka-console-consumer --bootstrap-server localhost:9092 --topic golang.monitoring.system.metrics --from-beginning

• Query Service:

      curl -X GET http://localhost:8083/qs?service_name="app_service"&metric_name="cpu"&time_interval=60

References

1. https://www.statcan.gc.ca/en/data-science/network/monitoring-alerting-system
2. https://systemdesignschool.io/problems/realtime-monitoring-system/solution