Talos OS: Persistent Volumes with Rook Ceph

Introduction

In the previous post I got a 3-node Talos cluster running, but it has no storage - anything that writes to disk loses everything the moment its pod moves. The reason I care is Postgres, which is coming in the next post, so this one is about getting a real, replicated StorageClass in place first: Rook Ceph, spread across all three nodes.

Prerequisites

The 3-node Talos cluster from the previous post.
Helm 3.
A spare, unpartitioned chunk of disk on each node for Ceph to use.

What Ceph and Rook Actually Do

Without something like this, a PersistentVolumeClaim backed by local disk is stuck on whichever node created it - if the pod gets rescheduled to another node, its data doesn’t come with it. Ceph pools the disks across all three nodes into one distributed storage cluster and replicates data between them, so a volume isn’t tied to a single node’s disk anymore - a pod can land on any of the three and still mount the same volume, with Ceph handling where the actual replicas live and keeping them in sync.

Rook is what makes running Ceph itself not miserable: it’s a Kubernetes operator that deploys and manages the Ceph cluster - mons, mgrs, OSDs, the CSI driver - from a handful of CRDs, and reacts to things like a node going down by rebalancing data instead of me doing it by hand.

Giving Talos a Raw Partition for Ceph

Each node only has one NVMe drive, so instead of a second physical disk I had Talos carve the drive into two managed volumes: the normal EPHEMERAL volume for the OS, and a raw volume for Ceph to claim as a block device.

yaml
apiVersion: v1alpha1
kind: VolumeConfig
name: EPHEMERAL
provisioning:
  diskSelector:
    match: disk.transport == 'nvme'
  maxSize: 100GiB
---
apiVersion: v1alpha1
kind: RawVolumeConfig
name: osd-data
provisioning:
  diskSelector:
    match: disk.transport == 'nvme'
  minSize: 10GiB
  grow: true

EPHEMERAL is where container images, pod logs, and etcd actually live - the normal day-to-day disk usage of running Kubernetes on the node. I capped it at 100GiB on purpose: left unbounded, that’s the volume that grows quietly over time as images pile up and logs accumulate, and I don’t want it eating into the space I need for Ceph. osd-data gets a 10GiB floor and grow: true to claim whatever’s left on the disk beyond that cap.

Applying that config (same talosctl apply-config flow as part 1) makes Talos create the partition and label it r-osd-data, which is what I point Rook at below.

Installing Rook Ceph

I install the upstream rook-ceph and rook-ceph-cluster charts with a values file. The important parts:

yaml
rook-ceph-cluster:
  toolbox:
    enabled: true   # for `ceph status` debugging

  cephClusterSpec:
    mon:
      count: 3
      allowMultiplePerNode: false
    mgr:
      count: 2   # active/standby
    dashboard:
      enabled: true
    storage:
      useAllNodes: false
      useAllDevices: false
      nodes:
        - name: "node1"
          devices:
            - fullpath: /dev/disk/by-partlabel/r-osd-data
        - name: "node2"
          devices:
            - fullpath: /dev/disk/by-partlabel/r-osd-data
        - name: "node3"
          devices:
            - fullpath: /dev/disk/by-partlabel/r-osd-data

shell
❯ helm repo add rook-release https://charts.rook.io/release
❯ helm repo update
❯ helm install rook-ceph rook-release/rook-ceph -n rook-ceph --create-namespace
❯ helm install rook-ceph-cluster rook-release/rook-ceph-cluster -n rook-ceph -f values.yaml

The operator (rook-ceph) and the cluster (rook-ceph-cluster) are two separate chart installs into the same namespace - the operator first, then the cluster chart with the values above, which is what actually stands up the mons, mgrs, and OSDs.

3 nodes means 3 mons, which is the smallest quorum that tolerates one mon going down. useAllDevices: false plus an explicit nodes: list matters here - without it, Rook will happily try to claim any unformatted disk it finds, including the wrong one.

Replication Across All Three Nodes

yaml
  cephBlockPools:
    - name: replicapool
      spec:
        failureDomain: host
        replicated:
          size: 3
          requireSafeReplicaSize: true
      storageClass:
        enabled: true
        name: ceph-rbd
        isDefault: true
        reclaimPolicy: Delete
        allowVolumeExpansion: true

failureDomain: host plus size: 3 means every object gets a full copy on each of the three nodes - I can lose one node entirely and still have every piece of data available twice over. ceph-rbd becomes the default StorageClass, so anything that creates a PersistentVolumeClaim without specifying one lands here automatically.

I also explicitly turned off the two other pool types the chart wants to create by default:

yaml
  cephFileSystems: []
  cephObjectStores: []

Both CephFilesystem and CephObjectStore default to erasure-coded pools that need 3 failure domains of their own on top of the block pool. I only need RBD block storage for a database, so leaving those enabled would just be wasted capacity on a 3-node cluster.

Testing It With a Throwaway Pod

Before trusting it with a database, I wanted to prove a pod’s data actually survives a restart:

yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-test-pvc
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: ceph-rbd
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: ceph-test-pod
spec:
  containers:
    - name: test
      image: busybox
      command: ["sh", "-c", "echo hello from ceph > /data/hello.txt && sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: ceph-test-pvc

shell
❯ kubectl apply -f ceph-test.yaml
❯ kubectl get pvc ceph-test-pvc
NAME            STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS
ceph-test-pvc   Bound    pvc-a1b2   1Gi        RWO            ceph-rbd

❯ kubectl delete pod ceph-test-pod
❯ kubectl apply -f ceph-test.yaml
❯ kubectl exec ceph-test-pod -- cat /data/hello.txt
hello from ceph

The pod got deleted and recreated, the PVC stayed bound, and the file was still there. That’s the whole point.

Conclusion

Three Ceph mons, three-way replicated block storage, and a default StorageClass that survives a node dropping out. It’s not fast storage - network-replicated block storage on consumer NVMe over a home LAN never will be - but it’s durable, which is what a database actually needs. Next post: putting a real Postgres cluster on top of it.