# Mar 31, 2025

:::tip
*Resident in-response –* @[Dmitry Nourell](mention://a9cd11bb-2946-43d2-a27d-ecfb075c19ac/user/298a2e26-87de-4753-a045-717271c8b2f5) 

:::

## Teleport was unavailable for 12 hours.

In February 2025, @[Dmitry Nourell](mention://baa4138c-5d56-431d-a392-7441ff7ece73/user/298a2e26-87de-4753-a045-717271c8b2f5) introduced a limited number of split-horizon records on Uxie to avoid a LAN-WAN-LAN loop in firewall for hair-pinned connections. 

It was done using Query Forwarding mechanism in OPNsense's Unbound configuration, effectively redirecting all queries on `*.t.bksp.in` to the local resolver on the Ingress node. Unforeseen, this led to the case when the `_acme-challenge.*.t.bksp.in` record was also affected, and therefore, CoreDNS on Ingress served it.

When renewing a certificate, Caddy on Ingress tried to create a verification record using Cloudflare, but failed. In fact, the record was created, but Caddy failed to verify its existence because requests for `_acme-challenge.*.t.bksp.in` were mistakenly forwarded to our CoreDNS, instead of Cloudflare.

Nobody noticed this issue until the certificate finally expired.

### Action Points

- [x] Always use external DNS server for Caddy ACME challenges.
- [x] Discard Query Forwarding in favour of Overrides in Unbound.
- [x] Set up a proper external uptime monitor for services (Uptime Kuma?)
- [x] Explore possible ways to automatically check expiring certificates in Caddy without additional configuration (daemon, fetching active vhosts using Caddy API?)

---

**Documents**

- [BKSP 101](https://wiki.bksp.in/s/public/doc/bksp-101-HvrtlzB32F)
- [Проекты](https://wiki.bksp.in/s/public/doc/proekty-yjWZscgTvo)
- [Infrastructure](https://wiki.bksp.in/s/public/doc/infrastructure-VPKsJwVSVh)
- [Пространство](https://wiki.bksp.in/s/public/doc/prostranstvo-UF1eSbQDi5)
- [Equipment](https://wiki.bksp.in/s/public/doc/equipment-t5MYAVJ8l0)
- [Other](https://wiki.bksp.in/s/public/doc/other-uPW1YJQaRt)