Mar 31, 2025

Resident in-response – @Dmitry Nourell

Teleport was unavailable for 12 hours.

In February 2025, @Dmitry Nourell introduced a limited number of split-horizon records on Uxie to avoid a LAN-WAN-LAN loop in firewall for hair-pinned connections.

It was done using Query Forwarding mechanism in OPNsense's Unbound configuration, effectively redirecting all queries on *.t.bksp.in to the local resolver on the Ingress node. Unforeseen, this led to the case when the _acme-challenge.*.t.bksp.in record was also affected, and therefore, CoreDNS on Ingress served it.

When renewing a certificate, Caddy on Ingress tried to create a verification record using Cloudflare, but failed. In fact, the record was created, but Caddy failed to verify its existence because requests for _acme-challenge.*.t.bksp.in were mistakenly forwarded to our CoreDNS, instead of Cloudflare.

Nobody noticed this issue until the certificate finally expired.

Action Points

  • Always use external DNS server for Caddy ACME challenges.

  • Discard Query Forwarding in favour of Overrides in Unbound.

  • Set up a proper external uptime monitor for services (Uptime Kuma?)

  • Explore possible ways to automatically check expiring certificates in Caddy without additional configuration (daemon, fetching active vhosts using Caddy API?)