Observability on a budget
You can't run a small, efficient platform safely if you can't see what it's doing. Here's how we get full-stack visibility without a heavyweight monitoring bill.
Three pillars, lightweight tools
Our monitoring stack is chosen to be resource-frugal — which matters when the whole point is a small, efficient cluster — while still giving us metrics, logs, and alerts across every service.
Victoria Metrics
A Prometheus-compatible time-series database that's notably lighter on memory and disk, with long-term retention for trend analysis.
Grafana
Dashboards over both metrics and logs, so service health, traffic, and errors are visible at a glance.
Loki
Log aggregation that indexes labels rather than full text — cheap to run, and correlated with metrics in the same dashboards.
Collection runs everywhere
Lightweight agents run as DaemonSets across every node — including the control plane — to scrape metrics, ship logs, and export node-level statistics. Running them everywhere means there are no blind spots: if a process is consuming resources or a node is under pressure, we see it.
Alerts that reach a human
Metrics are only useful if the right alert reaches the right person. Alerting rules feed an Alertmanager that routes notifications to the appropriate team chat channel by severity and domain — infrastructure issues, production application alerts, and pre-production alerts each go where they belong. Node-level lifecycle events — like a pending reboot — are announced too, so nothing happens silently.
Dogfooding: live metrics on this very site
The request counts on our home page aren't decoration — they're real figures pulled live from the same metrics database that monitors production. It's a small example of a bigger principle: the platform is observable enough that we can safely expose a slice of it to the public. That same visibility is what lets us run zero-downtime deployments with confidence and keep the cluster small and efficient.