Jul 08, 2026
RiR: @@Leonid Evdokimov x @Dmitry Nourell
BKSP ID & Teleport were down for 19h
Root cause: lack of maintenance (rebalance) on Beartic’s btrfs root led to free chunk exhaustion, rendering the entire pool read-only. Without write access, both BKSP ID (Logto) and Teleport were unable to create a new authentication sessions.
Fix: @Dmitry Nourell made a Hail Mary move by adding a tmpfs-backed loopback device to a root pool, adding an another free chunk to let btrfs properly rebalance. Never try this at home.
Action Points:
@Dmitry Nourell added a
btrfs-maintenanceservice on Beartic to perform weekly rebalance and monthly scrub.Discard further use of btrfs on Infrastructure nodes as a go-to File System.
End2space discarded some of residents’ profiles
TBD by @Leonid Evdokimov