Tech-kutak
Sign in Subscribe

Building tomorrow, one prompt at a time

Practical Homelab Automation: Self-Hosted CI/CD, IaC, and Observability

Ivan Horvatić 7 min read
Practical Homelab Automation: Self-Hosted CI/CD, IaC, and Observability

If your homelab feels like a collection of separate utilities, you’re not shipping software, you’re stack-building chaos. The fix is to give your environment an automation spine: a lean CI/CD pipeline, a repeatable infrastructure workflow, and a watchful eye on health. This article is a blueprint for a pragmatic, maintainable setup you can actually maintain over years, not months. It’s not about chasing shiny tech; it’s about boring, reliable, repeatable processes that let you deploy your own projects with confidence.

Why bother with a self-hosted stack in a homelab?

  • Privacy and control. You own your data and your pipelines; you don’t rely on a dozen SaaS services you barely understand.
  • Speed and local air-gapping. Your CI runs locally, builds are cached locally, and you’re not fighting per-commit latency from third parties.
  • Learning by doing. You’ll touch Terraform, Ansible, Kubernetes (or at least k3s), Docker, and Prometheus—the stack you’ll actually use in production roles.

Below is a practical, opinionated path to build a compact, maintainable self-hosted CI/CD and observability stack in a typical homelab.

Design principles I actually use

  • Start small, instrument everything, and automate the boring bits first. The first pipelines should be “build, test, publish” for a single repo; don’t chase multi-repo microservices yet.
  • Separate concerns. One host for Git and CI, one for the cluster or apps, one for monitoring/logging. Don’t shove everything onto one box unless you have to.
  • Use IaC everywhere. Terraform for infrastructure, Ansible for configuration, and a small amount of code that’s easy to reason about.
  • Use lightweight, maintainable components. K3s instead of a full Kubernetes plane; Drone CI instead of GitLab CE if you don’t need the extra features; Prometheus + Grafana + Loki for observability (not every feature, just what you actually use).

A concrete stack to start with

  • Hypervisor: Proxmox VE (or similar). It gives you cheap hardware-backed virtualization with snapshots and backups.
  • VMs or containers:
  • Git + CI server: Gitea for Git with Drone CI for pipelines (Drone integrates well with Gitea without bloating you with GitLab’s heft).
  • App cluster: Lightweight Kubernetes (k3s) or a Docker Compose-based runtime for services you’re building.
  • Observability: Prometheus + Grafana for metrics; Loki (or Loki-like) for logs; Alertmanager for alerts.
  • Reverse proxy and TLS: Traefik or Nginx with Let's Encrypt.
  • IaC and automation: Terraform (libvirt or local provider for virtualization), Ansible for provisioning, and a small bash/Makefile workflow for glue code.

Setting up the foundation (high level)

  • Host a dedicated, resilient management network. A separate VLAN or at least a dedicated NIC for management will save you headaches.
  • Storage matters more than you think. Use ZFS if possible, or at least separate disks for OS and data. Snapshots and backups are not optional.
  • Regular backups of VMs and critical config. Keep offsite or offline copies of the IaC definitions so you can rebuild in a pinch.
  • Patch cadence. Don’t chase every CVE; establish a weekly cycle to update OS and critical components, test in a staging VM, and roll out.

Step-by-step plan you can follow this weekend

1) Set up the hypervisor

  • Install Proxmox VE on your hardware. Enable backups, enable a separate datastore for VM disks, and set up a basic firewall.
  • Create a small, fast VM for Gitea + Drone, a larger node for k3s cluster, and a logging/monitoring VM.
  • Ensure you have a static IP plan, a DNS entry, and TLS ready for the services you’ll expose.

2) Build the Git + CI host (Gitea + Drone)

  • Install Gitea. It’s simple and fast for a self-hosted Git server.
  • Install Drone CI server and its runner, wired to Gitea. Drone is a great choice here because it’s lightweight and easy to secure.

3) Create a tiny Kubernetes cluster (k3s)

  • Install k3s on a VM (or a couple of VMs if you want a real multi-node cluster). Use a simple single-node setup for the start; you can scale later.
  • Optional: Install a small internal container registry (not necessary at first, but helpful as you push images from Drone).

4) Set up observability

  • Deploy Prometheus for metrics, Grafana for dashboards, and Loki for logs. Start with node exporters and a few app-specific metrics endpoints.
  • Create a few dashboards that answer a few key questions: “Is my app healthy?”, “What’s the deployment cadence?”, “Which pipelines fail and why?”
  • Wire Alertmanager to notify you on failures or high-latency events.

5) Automate provisioning with Terraform and Ansible

  • Use Terraform to provision the VMs (or the VMs’ networking). If you’re using libvirt, the Terraform libvirt provider can do the VM lifecycle.
  • Use Ansible to bootstrap the VMs: install Docker, deploy Drone, install k3s, deploy Prometheus/Grafana/Loki, and configure Traefik.

6) Build an initial CI/CD pipeline

  • Create a sample repo in Gitea and wire it to Drone.
  • Add a .drone.yml with a simple build/test/publish flow. You’ll iterate on this as you learn what to optimize.

A practical .drone.yml example (for a simple Python project)

# .drone.yml
kind: pipeline
name: default

steps:
- name: build
image: docker:20.10
commands:
- docker build -t your-registry/your-app:${DRONE_COMMIT_SHA} .
- docker push your-registry/your-app:${DRONE_COMMIT_SHA}

- name: test
image: python:3.11
commands:
- pip install -r requirements-dev.txt
- pytest -q

- name: release
image: plugins/docker
settings:
repo: your-registry/your-app
tags: ${DRONE_COMMIT_SHA},latest

Note:

  • This assumes you have a private registry accessible by Drone and your nodes. If you don’t, start with building and testing in place, then add a registry later.
  • The pipeline is intentionally simple at first. The point is to have a real pipeline that you can watch, fail fast, and learn from.

A minimal Ansible bootstrap (example)

  • Hosts: inventory.ini with [ci] (Drone + Gitea), [cluster] (k3s nodes), [monitoring] (Prometheus/Grafana)
  • Playbook fragment to install Docker and Docker Compose, then deploy Drone and Gitea
- hosts: ci
become: true
tasks:
- name: Install Docker
apt:
name: docker.io
state: present
update_cache: yes

- name: Install Docker Compose
get_url:
url: https://github.com/docker/compose/releases/download/2.5.0/docker-compose-$(uname -s)-$(uname -m)
dest: /usr/local/bin/docker-compose
mode: '0755'

A lightweight Terraform sketch (libvirt provider)

provider "libvirt" {
uri = "qemu:///system"
}

resource "libvirt_domain" "git_ci" {
name   = "git-ci"
memory = "4096"
vcpu   = 2

disk {
volume_id = libvirt_volume.disk.id
}

network_interface {
network_name = "default"
}

graphics {
type = "spice"
}
}

That Terraform snippet is intentionally tiny. The point is to show you can codify VM lifecycles, then use Ansible to bootstrap inside the VM. If you want to automate from a single box, you can also use Docker Compose to run Drone + Gitea in containers on one host. But I strongly recommend separation once you outgrow a single host.

Observability: what to monitor, what to alert

  • Metrics: CPU, memory, disk IO, network throughput, Pod/Container health, and application-level metrics. Start with node_exporter and cAdvisor (or kube-state-mistry in k3s).
  • Logs: Loki as a central log store, with a few dashboards for recent error trends and deployment failures.
  • Alerts: Alertmanager rules for high error rates in your pipelines, failed builds, or a push to the registry that didn’t go through.
  • Dashboards: A Grafana workspace with:
  • System health: CPU/memory/disk I/O
  • CI health: pipeline success rate, pipeline duration, queue length
  • App health: per-app error rates, latency
  • Ingress health: TLS termination, 5xx rate

Here’s a starter Prometheus scrape config for k3s nodes

global:
scrape_interval: 15s
scrape_configs:
- job_name: 'k3s'
static_configs:
- targets: ['localhost:9100']  # Node exporter

A few pragmatic tips from the trenches

  • Don’t over-engineer the pipeline. A single repo with a simple binary/artifact pipeline is enough to learn the flow. Complexity grows automatically as you bring more microservices in; resist adding more pipelines until you actually need them.
  • Use a single source of truth for credentials. HashiCorp Vault is great if you need it, but for many homelabs, a local secrets store (env vars, Docker secrets, or a minimal Vault) suffices. Do not hard-code secrets in code.
  • Automate backups early and test restores. The most painful thing is discovering your backups don’t work right before a disaster.
  • Keep your TLS termination at the edge. Let’s Encrypt with a reverse proxy simplifies cert management and reduces blast radius.
  • Document your playbooks and pipelines like you would production. The annoying part of a homelab is maintenance; good docs prevent you from breaking your own system every few months.

Security and hardening notes

  • Regularly patch OS and container runtimes. If you’re on Proxmox with VMs, you still need to patch the VMs themselves; automation helps here, but manual oversight remains valuable.
  • Use non-root users for SSH and CI agents where possible.
  • Apply network policies if you’re using Kubernetes (even on k3s). A few simple allow-listed endpoints reduce blast radius.
  • Limit public exposure of the Git and CI endpoints. If you can, keep them on a private network with a reverse proxy exposing only TLS endpoints to the outside.

Concrete benefits you’ll notice

  • Faster iteration: you can push changes to a project and see a pipeline run locally and reliably, without waiting for a cloud runner to respond.
  • Consistency: IaC makes your environment reproducible. If you move hardware or reset a VM, you can recreate your entire stack with a few commands.
  • Visibility: a centralized observability stack makes it obvious when a pipeline fails or when a node health problem arises.

Common pitfalls and how to dodge them

  • Pitfall: “It’s all in my head.” Don’t rely on manual dashboards or memory of what’s deployed. Codify your reality.

Solution: Automate the provisioning and create dashboards that reflect your real state. Make the dashboards low-friction to update as you add services.

  • Pitfall: Over-optimizing early.

Solution: Start with a minimal, working pipeline. Expand capabilities only after you’re comfortable with the basics.

  • Pitfall: One big, single point of failure.

Solution: Add a little redundancy: a second drone runner, a second k3s node, or a second monitoring sink. It pays off when you wake up to a failed service.

Actionable conclusion

  • Start with a 2-VM baseline: one for Gitea + Drone, one for k3s cluster. Get a simple pipeline building a tiny project and pushing a container image to a private registry.
  • Add monitoring in seconds: install Prometheus + Grafana, wire a basic node_exporter, and craft a single dashboard that answers “is this thing healthy?”.
  • Expand gradually: add Loki for logs, add a second node to the cluster, and bring in Terraform + Ansible for repeatable provisioning. Don’t skip the code-first mindset: every change should be codified and auditable.
  • Ship something you can support. If your pipeline isn’t maintainable or your docs are absent, you’ll abandon the stack in six months. Make it boring, repeatable, and boringly reliable.

If you follow this approach, you’ll end up with a homelab that isn’t a curiosity or a toy, but a practical platform you can rely on for personal projects, side hustles, and learning. The automation spine you build here will be the same spine you encounter in real-world jobs—so you’ll be not just tinkering, but growing practical capabilities you can take to paid work or open-source contributions.

Happy automating. Start small, stay pragmatic, and build a stack you can actually operate for years.