NOTE: This repository is currently in a state of flux as I finalize details of my cluster and slowly both learn and also move to different architectural patterns. In particular, the Helm and Terraform files will likely be drastically updated later as I migrate files and (eventually) bring Atlantis online for applying Terraform changes.

GitOps

🔎 About

This repository contains ArgoCD, Helm, and Terraform files for declarative deployments with Kubernetes, specifically k3s.

You can use these files to stand up your own on-prem Kubernetes cluster. While this repository was built to be run on Raspberry Pi devices, it should be equally valid anywhere Kubernetes can run.

If you want to implement this for yourself, please follow the setup document (which is actively being updated).

🎖️ Features

• App-of-apps: A root Argo CD Application deployment schema which recursively manages child apps
• Namespace deployments: argocd, cert-manager, kube-system, logging, longhorn-system, monitoring, and applications-eng
• Cert-manager: X.509 certificate management for Kubernetes
• Longhorn: Distributed on-prem file storage with multiple storage classes
• Metal LB: An on-prem native software load balancer
• Monitoring: Prometheus Operator with Grafana using storage PVC through Longhorn
• n8n: Workflow automation platform with persistent storage
• Ollama: Runtime for AI models on a GPU node
• Dashboard UI for:
  • Argo CD: For controlling deployments and rollbacks
  • Grafana: For building dashboards against Prometheus data
  • Longhorn: For controlling the distributed block storage setup
  • n8n: For creating and managing automated workflows
  • Prometheus: For querying against raw data from pods/nodes/deployment resources

🧱 Project Management

Work for this repository is housed in this Trello board.

📁 Project Structure

├── argocd/                                      # ArgoCD application definitions
│   ├── apps/                                    # Application-level deployments
│   │   ├── applications/                        #
│   │   ├── argocd/                              #
│   │   ├── cert-manager/                        #
│   │   ├── kube-system/                         #
│   │   ├── logging/                             #
│   │   ├── longhorn-system/                     #
│   │   └── monitoring/                          #
│   ├── namespaces/                              # Namespace-level deployments
│   └── root/                                    # Root ArgoCD application deployment
├── helm/                                        # Helm charts
│   ├── argocd/                                  #
│   ├── cert-manager/                            #
│   ├── grafana/                                 #
│   ├── longhorn/                                #
│   ├── n8n/                                     #
│   ├── nvidia-device-plugin/                    #
│   ├── ollama-app/                              # vLLM doesn't support 5090 FE cards right now, so Ollama it is
│   ├── prometheus/                              #
│   ├── prometheus-operator/                     #
│   ├── prometheus-service-monitors/             #
└── terraform/                                   # Terraform configurations
    ├── namespaces.tf                            # Kubernetes namespace definitions
    ├── provider.tf                              # Terraform provider configuration
    └── storage-classes.tf                       # Longhorn storage class definitions

🛠️ Built With

Hardware

The cluster this repo's files runs on uses Raspberry Pi 5 devices, specifically the 16gb version.

Here's the hardware list of what each of the control/worker nodes is using:

1. Raspberry Pi 5
2. NVMe + POE+ Pi 5 Hat and Active Cooler
3. Samsung 2TB NVMe SSD
4. 256gb Micro SD Card

It's worth noting that one of my nodes is a computer running Ubuntu with a nice GPU, but that's really outside the scope of any guides I'd give for deploying this repository. The only part of this that will impact you is any apps that have node affinity for that setup (like the nvidia-device-plugin-app and ollama-app deployments), but you can easily remove that from your own deployments.

The rest of the nodes are Raspberry Pi 5s as described above.

Software

• Argo CD
• Cert Manager
• Grafana
• Grafana Loki
• Grafana Promtail
• Helm
• Kubernetes, specifically K3s
• Longhorn
• Metal LB
• n8n
• Nvidia Container Toolkit
• Ollama (vLLM does not support RTX 5090 FE cards right now)
• OpenFaaS (coming soon)
• Prometheus and Prometheus Operator
• Sealed Secrets
• Terraform
• Traefik

🙇🏻‍♂️ Acknowledgements

• Edede Oiwoh for inspiring me to build a home cluster and for bouncing ideas around
• rpi4cluster.com for tips on GitOps with Raspberry Pi setups (even if the notes weren't current and Helm/Argo configurations weren't file-based)
• Twitter (now X), Loom, and Tesla for teaching me proper GitOps processes and giving me a chance to move mountains with them
• gitops-patterns repository for what will likely be ongoing sources of truth for modern architecture patterns