Fun experience (homelab)

I think this deserves more attention than anything that I’ve done commercially, because this is fun and cool! C:

Why?

At some point of my life I decided that I’m gonna build my own intranet. I’ve had that urge since the day I have retired my laptop (and made it into a debian server), but that urge has just grown over the years.

Beginnings

Last year I got 3 mini pcs and decided to start small - 3 baremetal fedora servers running nomad(+consul) for workload orchestration, because, and even though I’ve had much more experience with k8s, I thought it was overkill. I have intentionally decided not to expose my homelab to public internet, and I’m keeping it behind tailscale.

So, you might think, you’ve solved compute, what about storage? Well, at first I tried to set up ceph myself. I used rbd for storage then. It was partially successful, but then I decided to migrate my entire network from 192.168.1.0/24 to 192.168.42.0/24, and everything broke beyond repair. I did not have much experience with ceph, so after a couple of sleepless nights, I just dropped it and took glusterfs. But moving to gluster has made provisioning of volumes and reproducibility of the setup suffer a lot, so after a while I came back to ceph (but this time around orchestrated by proxmox).

Second iteration

I decided to stick with nomad, but deploy proxmox and run nomad in vms. All of this of course would be managed through terraform. Proxmox is a good orchestrator for ceph, so I hoped it would be easier this time around. And it was! - well, kinda, if you overlook the struggles with setting up the csi plugin with almost no docs. But this has worked for a while, so at some point I decided to start moving all of the services I use to my homelab.

That has revealed the limitations of both nomad and ceph rbd.

  • I am not mentioning nomad’s problems with cross-job dependencies, and the issue that has been open for 10 years at this point.
  • Storage-wise rbd doesn’t support multiple writes, one should use cephfs for that.

So I decided to set up another CSI plugin - for cephfs this time around. After digging around in csi plugin’s codebase, I’ve finally figured out how to do that, but then I started hitting other issues with nomad and ceph having different states for the same volumes. At this point I decided to quit trying to make nomad work, and just gave up on it.

Last iteration

I have still had the vm module that I’ve created for nomad vms, so I just decided to tweak it, remove old vms, create new ones, take kubespray and get on with my life. Setting up cephfs csi was exteremely easy. At some point, I added another storage class for the hdd part of the setup.

As the result, my current setup is:

  • Baremetal infra: Reproducible setup through terraform.
  • Cluster: Kubernetes provisioned with Kubespray.
  • Ingress: Traefik v3 with Cloudflare DNS challenge.
  • Storage: Ceph via CSI drivers.
    • StorageClasses: csi-cephfs-sc (backed by ssds), csi-cephddfs-sc (backed by hdds).
  • App deployments: Everything done through helmcharts (existing or self-written).
  • A couple of rogue vms: Home assistant and minecraft servers are not the best thing to deploy to k8s, so each one of them has a vm (also managed through terraform).

I’m intending to make this setup opensource, but right now my git repo is scattered with secrets, so I need to fix that and observability first. I’ve had opentelemetry setup back when I was running nomad (with victorialogs and victoriametrics), but that doesn’t work anymore, and I’m looking into signoz.

I can show you all of this on a call if you are curious, whether you are a recruiter or an interested party, just drop me an email.

Last last iteration

Ok so I hate ansible, sorry. Tell me I’m wrong, but care I don’t.

So I decided to get some soy sauce and install tofu lsp on my nvim. I rewrote all the configuration I had into tofu that booted vms with talos (but still using same csi configuration with some additions). I love talos. It’s just so great. Couple of quirks and coredns redeploys later, I was a happy owner of a true HA setup at home. Oh and yes, this time around I went with proper tailscale operator and it’s tailscale ipv4 is mapped to A *.zemfira.me, so that all my devices in the tailnet work cool. Ah, and I still love both traefik and tailscale as you might’ve guessed. Now in tofu flavour!

This is heavily opinionated, but no more fucking yamls, if it gets really tough I just do something like that.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

resource "helm_release" "postgres_workloads" {
  name             = "postgres-workloads"
  namespace        = var.namespace
  chart            = "${path.module}/cnpg-workloads"
  version          = "1.0.0"
  create_namespace = true
  wait             = true
  timeout          = 3600

  values = [
    yamlencode({
      resources = [
        {
          apiVersion = "postgresql.cnpg.io/v1"
          kind       = "Cluster"
          metadata = {
            name      = local.cluster_name
            namespace = var.namespace
          }
          spec = {
            instances             = var.instances
            enableSuperuserAccess = true
            imageName             = "ghcr.io/cloudnative-pg/postgresql:18"
            storage = {
              size         = var.storage_size
              storageClass = "csi-rbd-sc"
            }
            postgresql = {
              parameters = {
                synchronous_commit = "off"
                wal_compression    = "on"
                max_connections    = local.max_db_connections
              }
            }
            monitoring = {
              enablePodMonitor = true
              customQueriesConfigMap = [
                {
                  name = "cnpg-custom-monitoring"
                  key  = "custom_metrics.yaml"
                }
              ]
            }
          }
        },
        {
          apiVersion = "postgresql.cnpg.io/v1"
          kind       = "Pooler"
          metadata = {
            name      = local.pooler_name
            namespace = var.namespace
          }
          spec = {
            cluster = {
              name = local.cluster_name
            }
            instances = 2
            type      = "rw"
            pgbouncer = {
              poolMode = "session"
              parameters = {
                max_client_conn      = "2000"
                default_pool_size    = "280"
                max_db_connections   = local.max_db_connections
                reserve_pool_size    = "25"
                reserve_pool_timeout = "3"
              }
              monitoring = {
                enabled = true
              }
            }
          }
        }
      ]
    })
  ]

  depends_on = [
    helm_release.cnpg_operator,
    kubernetes_manifest.cnpg_custom_monitoring,
  ]
}

I’m sorry, but I cannot write yaml, I have audhd and I’m a lowkey blind grandma, it’s not worth it, for the love of g-d. kyaml sounds fun, but go find tooling for that. Maybe I’m wrong. Email me, idk.

The rest is pretty cool, I use Bitwarden’s provider, store all secrets there, and then just setup all my providers and stuff based on that. As backend I use Neon (Postgres), and tofu has client side encryption, so I can happily hand out my data while holding onto passphrase.

I extended CSI a bit, cuz I’m learning a bit more about ceph, so now I have several pools (available thru csi mounts as storageclasses)

  • nvme rbd pool (RWO, so good for postgres, clickhouse, whatever block disks)
  • hdd rbd pool (RWO as well, seaweedfs is backed by this)
  • nvme cephfs pool (RWX pools, calibre books, songs)
  • hdd cephfs pool (RWX as well, films, series)

I know cache tier is kinda outdated in ceph, but I just have 2 nvmes sitting as hot cache for ALL of these pools.

Yes, you might say contention, but I might just reply 5 users monthly and avg of 250gb of cache set for each pool. DO YOU USE 250GB A DAY THAT OFTEN? LOOK ME IN THE EYE AND TELL ME YES, AND I’M GONNA JUDGE YOU VERY HARSHLY!

But anyway when I have spare money for proper SLC vs my current QLC ssds, I’ll consider it. And dare I say, I need even more memory, 16TiB ain’t nearly enough for proper communal infrasturcture. But that’s aight, we’ll get there.

Oh, and also I ain’t got no CI, because CI is a lie. You don’t need CI, you can just be happy. Just do something like this and live happily ever after.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

locals {
  obsidian_full_image_path = "${var.registry_host}/obsidian-headless"
}

resource "terraform_data" "obsidian_build" {
  triggers_replace = [
    var.registry_host,
    var.image_tag,
    filesha1("${path.module}/Containerfile"),
  ]

  provisioner "local-exec" {
    command = <<-EOT
      podman build -t '${local.obsidian_full_image_path}' "${abspath(path.module)}"
      podman push '${local.obsidian_full_image_path}'
    EOT
  }
}

This is the last last version of my infra, I think I’m pretty settled, pero vamos a ver.