meatbag/infra

OpenTofu IaC for a 3-node Incus cluster running OCI containers.

Architecture

                           +------------------+
                           |    Internet      |
                           +--------+---------+
                                    |
                              port forward
                                    |
                           +--------+---------+
                           |   UDM Router     |
                           | DHCP, DNS, NAT   |
                           +--+------------+--+
                              |            |
              ----------------+--        --+----------------
              |  incusVLAN       |      |  homeARPA         |
              |  10.0.100.0/24   |      |  10.0.69.0/24     |
              +--+----+----+-----+      +----+----+---------+
                 |    |    |                 |    |
               kate kylie kari           nikki  mako

Nodes

 +--------------------------------------------------+
 |                    kate (leader)                  |
 |                   10.0.100.0/24                   |
 |                                                   |
 |  proxy:    caddy, endlessh                        |
 |  media:    sonarr, radarr, lidarr, bazarr,        |
 |            prowlarr, sabnzbd, jellyfin,            |
 |            audiobookshelf, subsyt, bgutil          |
 |  web:      freshrss, homepage, dufs               |
 |  games:    valheim                                |
 |  forge:    forgejo, registry                      |
 |  monitoring: beszel, alloy                        |
 |  comms:    murmur                                 |
 |                                                   |
 |  ZFS: storage/incus, vault (2x4TB mirror)         |
 +--------------------------------------------------+

 +--------------------------------------------------+
 |                  kylie (CI)                       |
 |                 10.0.100.0/24                     |
 |                                                   |
 |  forge:      woodpecker, woodpecker-agent         |
 |  monitoring: loki, grafana                        |
 |                                                   |
 |  Podman: beszel-agent, podman-proxy (quadlet)     |
 +--------------------------------------------------+

 +--------------------------------------------------+
 |              kari (home automation)               |
 |                 10.0.100.0/24                     |
 |                                                   |
 |  home: homeassistant (VM), mosquitto, zigbee2mqtt |
 |                                                   |
 |  Podman: beszel-agent (quadlet)                   |
 +--------------------------------------------------+

 +--------------------------------------------------+
 |              nikki (management)                   |
 |                 10.0.69.0/24                      |
 |                                                   |
 |  Runs: just plan/apply/update (via SSH to kate)   |
 |  Has: incus remote configured, repo working copy  |
 +--------------------------------------------------+

Deployment flow

 nikki                        kate                      kari/kylie
 (working copy)               (tofu + state)            (remote nodes)
   |                            |                          |
   |  1. rsync working copy     |                          |
   |--------------------------->|                          |
   |                            |                          |
   |  2. ssh: tofu apply        |                          |
   |--------------------------->|                          |
   |                            |  3. incus create/update  |
   |                            |------------------------->|
   |                            |                          |
   |                            |  4. deploy-configs       |
   |                            |   (rsync + restart)      |
   |                            |------------------------->|
   |                            |                          |
   |                            |  5. deploy-machine       |
   |                            |   (apt, scripts, systemd)|
   |                            |------------------------->|

Repository structure

.
+-- terraform.tf          # Provider config, OCI registry remotes
+-- cluster.tf            # Cluster-wide Incus settings
+-- project.tf            # Instantiates all modules, passes variables
+-- variables.tf          # Image versions, MAC addresses, secrets
+-- machines.tf           # Triggers for machine config deployment
+-- terraform.tfvars      # Actual values (gitignored, contains secrets)
+-- justfile              # Command recipes
|
+-- modules/
|   +-- project/          # Utility: creates Incus project + base profile
|   +-- proxy/            # caddy, endlessh
|   +-- media/            # sonarr, radarr, lidarr, bazarr, prowlarr,
|   |                     # sabnzbd, jellyfin, audiobookshelf, subsyt, bgutil
|   +-- web/              # freshrss, homepage, dufs
|   +-- games/            # valheim
|   +-- forge/            # forgejo, registry, woodpecker, woodpecker-agent
|   +-- monitoring/       # beszel, loki, alloy, grafana
|   +-- comms/            # murmur
|   +-- home/             # homeassistant, mosquitto, zigbee2mqtt
|
+-- configs/              # Service config files (deployed via deploy-configs)
|   +-- caddy/            # Caddyfile
|   +-- alloy/            # config.alloy
|   +-- loki/             # local-config.yaml
|   +-- grafana/          # provisioning files
|   +-- mosquitto/        # mosquitto.conf
|   +-- subsyt/           # subscriptions.json
|
+-- machines/             # Host-level configs (deployed via deploy-machine)
|   +-- kate/             # (all hosts: beszel-agent binary + update timer)
|   |   +-- apt/          # sources + keyrings (debian, incus, opentofu) + packages.txt
|   |   +-- scripts/      # borg-backup, zfs-replicate, mirror-*-sync
|   |   +-- systemd/      # timers + services for above scripts
|   +-- kylie/
|   |   +-- apt/          # sources + keyrings (debian, incus) + packages.txt
|   |   +-- quadlet/      # podman-proxy
|   +-- kari/
|   |   +-- apt/          # sources + keyrings (debian, incus) + packages.txt
|   +-- nikki/            # arch fleet: pacman hooks + packages.txt, greetd;
|   +-- veronika/         # deployed from nikki (kate can't reach homeARPA)
|   +-- mako/
|
+-- scripts/
    +-- deploy-configs    # Syncs configs/ to hosts, restarts containers
    +-- deploy-machine    # Syncs machines/ to hosts (apt, scripts, systemd, quadlet)
    +-- check-updates     # Queries registries for new image digests

Commands

All commands run from the repo root on nikki:

just plan                          # tofu plan (syncs to kate first)
just apply                         # tofu apply
just check-updates                 # check registries for new image digests
just update                        # check-updates + taint changed images
just update-image <name> <module>  # force-taint and redeploy one image
just deploy-machine <host>         # deploy host configs (apt, scripts, systemd, quadlet)
just log <name> <project>          # view OCI container logs
just list-all                      # list instances across all projects
just query '{filter}'              # query Loki logs
just force-delete <name> <project> # unstick a crashed container

Adding a new service

1. Add incus_image + incus_instance + incus_storage_volume resources in the appropriate module
2. Add version variable to the module's variables.tf and root variables.tf
3. Wire the variable through project.tf
4. If the service has config files: add to configs/<service>/, add entry in scripts/deploy-configs, add terraform_data trigger resource
5. Add image entry to scripts/check-updates IMAGES array

Adding a new machine config

1. Create files under machines/<host>/<type>/ (apt, pacman, scripts, systemd, quadlet, greetd)
2. Cluster hosts only: add a terraform_data trigger in machines.tf watching the fileset
3. Run just deploy-machine <host> or (cluster hosts) let just apply pick it up

The arch fleet (nikki, veronika, mako) is deliberately absent from machines.tf: tofu runs on kate, and kate cannot reach homeARPA. Those hosts are deployed straight from nikki via just deploy-machine <host>. System-level config for them lives here; chezmoi manages only $HOME.