Skip to content

Architecture

How thomelab is built and operated. For what it runs, see Overview.

Clusters

Cluster Role Cluster DNS domain Status
thomelab Primary cluster. Most self-hosted apps run here. cluster.local Active
public Internet-facing edge: Traefik, Netbird reverse proxy, Keycloak SSO, a small number of public apps. public.local v1 in production, v2 in preparation

Repositories

Each repository holds declarative config. ArgoCD applies Kubernetes manifests; Ansible and OpenTofu apply everything below Kubernetes. No repository takes direct commits to main — every change is a GitLab issue, a branch named after it, and a merge request.

Repository Purpose Visibility
thomelab-infra Kubernetes manifests (Kustomize + ArgoCD) for both clusters. Public
thomelab-ops Ansible and OpenTofu: OS maintenance, cluster lifecycle, resource management (Keycloak realms, Netbird groups/policies, DNS records, S3 buckets). Private
thomelab-bootstrap-infra Bootstraps the layer below Kubernetes, mainly the Talos clusters today Private
thomelab-site Source for thomelab.net: homepage and this docs site (public and private builds). Private

GitOps flow

Each cluster's Kubernetes state is one ArgoCD app-of-apps in thomelab-infra, in two layers:

  • cluster-tools/ — the tools a cluster needs to run anything else: ArgoCD, Traefik, cert-manager, Sealed Secrets / External Secrets, storage provisioners, Netbird.
  • applications/ — the self-hosted apps, each as base/ (the ArgoCD Application) plus overlays/<cluster>/ (cluster-specific patches and secrets).

ArgoCD watches main and auto-syncs. To test a change before merging, a temporary ArgoCD Application points at the feature branch, is validated, then deleted. Manifests are never applied from a branch directly.

Networking and exposure

Public traffic targets thomelab.net. Two systems route it:

  • Traefik, on the public cluster, for apps with an explicit IngressRoute: ntfy, Vaultwarden, Gatus, Keycloak, the homepage/docs site, ArgoCD's dashboards.
  • Netbird's reverse proxy, for everything else — apps registered as netbird_reverse_proxy_service entries in thomelab-ops. Traefik forwards unrecognized hosts to Netbird over a TLS passthrough route; Netbird terminates TLS itself (ACME, tls-alpn-01) per registered subdomain.

A subdomain registered in neither place fails the TLS handshake rather than redirecting anywhere. Fixing this requires a wildcard certificate (DNS-01, not currently configured) and would require listing every Netbird domain in Traefik as well; left unfixed as a known limitation.

SSO is Keycloak, via OIDC (for applications supporting it), oauth2-proxy (for apps on the public cluster without oidc support), and behind Netbird's authentication (for apps on thomelab that don't support OIDC).

Cluster and admin access

Administrative access (kubectl, talosctl) is kept separate from the app-traffic path above. public's host firewall permits only port 443 by default; the Kubernetes and Talos API ports are opened only temporarily, by hand, when genuinely needed. Day to day, both clusters' API servers are reached through Tailscale's Kubernetes-operator API-server proxy rather than a direct connection — a direct-connection kubeconfig context still exists as an explicit, non-default fallback.

Netbird and Tailscale overlap in the mesh-networking sense but do different jobs here: Netbird carries the app-traffic reverse proxy and general pod-to-pod mesh access described above; Tailscale exists specifically for API-server access to the clusters.

Secrets and encryption

  • sops + age encrypts sensitive files in thomelab-bootstrap-infra (which retired git-crypt in favor of it) and protects OpenTofu backend/state secrets in thomelab-ops.
  • git-crypt still encrypts a narrower set of files in thomelab-ops — local-only inventory overrides and per-module OpenTofu secrets not covered by sops.
  • ansible-vault additionally protects thomelab-ops/inventory/hosts.yaml; the vault password is kept outside git, in ~/.vaultpass.
  • In Kubernetes: Sealed Secrets is the mechanism for committing encrypted manifests on the public cluster, and for apps on thomelab that can't rely on OpenBao. The default mechanism for thomelab is External Secrets + OpenBao.

Automation

Scheduled and triggered jobs run as Argo Workflows on thomelab. Renovate runs continuously on thomelab, watching all repositories and opening merge requests for dependency and container digest bumps; some auto-merge.

Backups

Velero backs up Kubernetes-level resources on both clusters. It does not cover everything: local-path PersistentVolumes are not captured by Velero's PodVolumeBackup the way Longhorn volumes are. CNPG barman (Postgres) and rclone (everything else) run alongside Velero to close that gap.

Further detail

Each repository's own README.md covers its specific commands and conventions.