meatbag/infra

OpenTofu IaC for a 3-node Incus cluster running OCI containers, plus the host config of its nodes.

Architecture

                           +------------------+
                           |    Internet      |
                           +--------+---------+
                                    |
                              port forward
                                    |
                           +--------+---------+
                           |   UDM Router     |
                           | DHCP, DNS, NAT   |
                           +--+------------+--+
                              |            |
              ----------------+--        --+------------------------
              |  incusVLAN       |      |  homeARPA                 |
              |  10.0.100.0/24   |      |  10.0.69.0/24             |
              +--+----+----+-----+      +----+-------+-----+------+-+
                 |    |    |                 |       |     |      |
               kate kylie kari           nikki veronika  mako jessika

The firewall is one-way: homeARPA can reach incusVLAN, but incusVLAN can't open connections to homeARPA. The homeARPA hosts are managed from meatbag/fleet, not this repo.

Nodes

 +---------------------------------------------------+
 |              kate (Incus DB leader)               |
 |                                                   |
 |  proxy:      caddy, endlessh                      |
 |  media:      sonarr, radarr, lidarr, bazarr,      |
 |              prowlarr, sabnzbd, jellyfin,         |
 |              audiobookshelf, subsyt, bgutil,      |
 |              compost                              |
 |  photos:     immich, immich-ml, immich-postgres,  |
 |              immich-redis, immich-exif, elodie    |
 |  web:        freshrss, homepage, dufs,            |
 |              vaultwarden, claudia-tama            |
 |  games:      valheim                              |
 |  forge:      forgejo, registry                    |
 |  monitoring: beszel, alloy-kate                   |
 |  comms:      ergo, murmur                         |
 |                                                   |
 |  Host: tofu + state, infra-reconcile, borg and    |
 |  ZFS backups, apt mirrors, fail2ban, nftables     |
 |  ZFS: storage/incus, vault                        |
 +---------------------------------------------------+

 +---------------------------------------------------+
 |                    kylie (CI)                     |
 |                                                   |
 |  forge:      woodpecker, ci (VM)                  |
 |  monitoring: loki, grafana, alloy-kylie           |
 |  fitness:    spis, spis-resolver-llm, vitals      |
 |  comms:      smtp                                 |
 |                                                   |
 |  Host: ci VM firewall (nftables)                  |
 +---------------------------------------------------+

 +---------------------------------------------------+
 |              kari (home automation)               |
 |                                                   |
 |  home:       homeassistant (VM), mosquitto,       |
 |              zigbee2mqtt                          |
 |  monitoring: alloy-kari                           |
 +---------------------------------------------------+

 All three run beszel-agent (plain binary + systemd).

 +---------------------------------------------------+
 |              nikki (operator seat)                |
 |                 10.0.69.0/24                      |
 |                                                   |
 |  Runs: just plan/apply/status (tofu runs on kate) |
 |  Has: incus remote, repo working copy, sops key   |
 +---------------------------------------------------+

Deployment flow

Instances, images and volumes are tofu, applied from nikki:

 nikki                        kate                      kari/kylie
 (working copy)               (tofu + state)            (remote nodes)
   |                            |                          |
   |  1. rsync working copy     |                          |
   |--------------------------->|                          |
   |                            |                          |
   |  2. ssh: tofu apply        |                          |
   |     (secrets decrypted     |                          |
   |      in RAM on kate)       |                          |
   |--------------------------->|                          |
   |                            |  3. incus create/update  |
   |                            |------------------------->|

Host and service config deploy from pushed main, not from just apply:

 forgejo (main)               kate                      kari/kylie
   |                            |                          |
   |  1. infra-reconcile.timer  |                          |
   |     fetch (every 5 min)    |                          |
   |--------------------------->|                          |
   |                            |  2. deploy-machine --all |
   |                            |   (apt, scripts, systemd,|
   |                            |    nftables, fail2ban)   |
   |                            |------------------------->|
   |                            |                          |
   |                            |  3. deploy-configs --all |
   |                            |   (rsync + restart)      |
   |                            |------------------------->|
   |                            |                          |
   |                            |  4. auto-apply           |
   |                            |   (digest bumps only)    |

Image versions are pinned in variables.tf. Renovate runs hourly from Woodpecker (.woodpecker/renovate.yaml) and opens PRs for them. A merged digest bump is applied by step 4. Anything else waits for just apply.

Repository structure

.
+-- terraform.tf          # Provider config, OCI registry remotes, state encryption
+-- cluster.tf            # Cluster-wide Incus settings
+-- project.tf            # Instantiates all modules, passes variables
+-- variables.tf          # Image versions (Renovate-annotated), MAC addresses, secrets
+-- backups.tf            # Secrets handed to kate's backup jobs
+-- beszel-agent.tf       # beszel-agent env file (hub key + token)
+-- secrets.sops.yaml     # Variable values, sops+age encrypted (see docs/secrets.md)
+-- renovate.json         # Renovate config
+-- justfile              # Command recipes
+-- mise.toml             # Local toolchain
|
+-- modules/
|   +-- project/          # Utility: creates Incus project + base profile
|   +-- proxy/            # caddy, endlessh
|   +-- media/            # sonarr, radarr, lidarr, bazarr, prowlarr, sabnzbd,
|   |                     # jellyfin, audiobookshelf, subsyt, bgutil, compost
|   +-- photos/           # immich (+ ml, postgres, redis, exif), elodie
|   +-- web/              # freshrss, homepage, dufs, vaultwarden, claudia-tama
|   +-- games/            # valheim
|   +-- forge/            # forgejo, registry, woodpecker, the ci VM
|   +-- monitoring/       # beszel, loki, grafana, alloy-{kate,kari,kylie}
|   +-- comms/            # ergo, murmur, smtp
|   +-- home/             # homeassistant, mosquitto, zigbee2mqtt
|   +-- fitness/          # spis, spis-resolver-llm, vitals
|
+-- configs/              # Service config files (deployed from main via deploy-configs)
|   +-- caddy/            # Caddyfile
|   +-- alloy-kate/       # config.alloy, one per node
|   +-- alloy-kari/
|   +-- alloy-kylie/
|   +-- loki/             # local-config.yaml
|   +-- grafana/          # provisioning files
|   +-- mosquitto/        # mosquitto.conf
|   +-- ergo/             # ircd.yaml
|   +-- immich-exif/      # rendered with a secret, so deployed by tofu
|
+-- machines/             # Host-level configs (deployed from main via deploy-machine)
|   +-- kate/             # REBUILD.md: bare-metal rebuild runbook
|   |   +-- apt/          # sources + keyrings (debian, incus, opentofu) + packages.txt
|   |   +-- scripts/      # infra-reconcile, borg-backup, zfs-replicate, mirror-*-sync, ...
|   |   +-- systemd/      # timers + services for the scripts above
|   |   +-- fail2ban/     # caddy-abuse jail
|   |   +-- nftables/     # blocklist
|   |   +-- journald/, smartd/, containers/, network/
|   +-- kylie/
|   |   +-- apt/          # sources + keyrings (debian, incus) + packages.txt
|   |   +-- nftables/     # ci-vm: what the CI VM can reach, by hostname (nft-resolve)
|   |   +-- scripts/, systemd/, smartd/, containers/
|   +-- kari/
|   |   +-- apt/          # sources + keyrings (debian, incus) + packages.txt
|   |   +-- scripts/, systemd/, smartd/, containers/
|
+-- scripts/
|   +-- deploy-machine    # Syncs machines/ to hosts (run by infra-reconcile)
|   +-- deploy-configs    # Syncs configs/ to hosts, restarts containers (same)
|   +-- auto-apply        # Applies merged image digest bumps (same)
|   +-- tofu-remote       # tofu on kate under with-secrets (behind just plan/apply)
|   +-- with-secrets      # Decrypts secrets.sops.yaml into RAM for one command
|   +-- status            # just status
|   +-- snapshot, deploy-certs, dr-drill, ...
|
+-- docs/                 # secrets, network, ipv6, mail, irc
+-- .woodpecker/          # Renovate cron pipeline

Commands

All commands run from the repo root on nikki:

just status [--quick]              # health + drift: git, cluster, mirrors, mail, Renovate PRs, units, backups, plan
just plan                          # tofu plan (syncs to kate first)
just apply                         # tofu apply
just reconcile                     # deploy machines/ + configs/ from pushed main now
just deploy-machine <host>...      # break-glass: deploy the working tree's machines/<host>
just log <name> [since] [limit]    # container stdout/stderr, from Loki
just restart <name>                # restart a container (project resolved from name)
just list-all                      # list instances across all projects
just query '{filter}'              # query Loki logs
just snapshot <service> [label]    # ZFS snapshot of a service's volumes, before a manual bump
just edit-secrets                  # edit secrets.sops.yaml
just force-delete <name>           # unstick a crashed container

just --list has the rest.

Adding a new service

1. Add incus_image + incus_instance + incus_storage_volume resources in the appropriate module
2. Add version variable to the module's variables.tf and root variables.tf. In the root one, put # renovate: image=<registry>/<repo> directly above default, and pin moving tags as tag@sha256:… so Renovate tracks it
3. Wire the variable through project.tf
4. If the service has config files: add to configs/<service>/ and add an entry in scripts/deploy-configs. Push before just apply creates the container, so the reconcile stages the config first (or run just reconcile)

Adding a new machine config

1. Create files under machines/<host>/<type>/ (apt, scripts, systemd, quadlet, fail2ban, nftables, …). Commit scripts executable: the mode comes from git
2. New host: add it to HOSTS in scripts/deploy-machine
3. Push. kate's infra-reconcile deploys it within 5 minutes, or run just reconcile

The Arch fleet (nikki, veronika, mako, jessika) lives in its own repo, meatbag/fleet: kate cannot reach homeARPA, so those hosts deploy from nikki.