When Ceph Backfill Was Not Actually Stuck

Recently I had a task that looked simple on paper: resize the root partitions on several Kubernetes nodes running a Rook-Ceph cluster.

The resize itself was not the most interesting part. The interesting part started after new OSDs appeared in the cluster and Ceph began rebalancing data. At that point the cluster looked healthy, but the rebalancing process almost stopped making progress. For more than 8 hours the number of misplaced objects stayed practically on the same figure, while most of the remapped PGs were sitting in backfill_wait.

Managing self-signed TLS for Docker Compose with step-ca

Almost a year ago, I set up a tiny homelab on an Intel NUC running Arch Linux. The original goal was fairly modest: run a few self-hosted services for personal use — Plex, Nextcloud, qBittorrent and Bitwarden.

I deliberately avoided Kubernetes. Even though k3s is perfectly capable of running on low-end hardware, I did not want to introduce another layer of complexity into a setup that was supposed to remain small and maintainable. Docker Compose felt more than enough for a single-node environment.