Mar 28, 2025

Resident in-response – @Dmitry Nourell

Inbound connections fail for 12 hours.

@Dmitry Nourell wrongfully configured NAT mapping, resulting in all ports being mapped to PBX NG LXC instead of their original destinations.

Due to this, Teleport traffic was also improperly routed, resulting in the inability to re-establish a remote connection to roll back those changes.

@Leonid Evdokimov accessed Uxie through one of the established tunnels and restored the valid configuration from the backup.

Root Cause

A heisenbug in the OPNsense Port Forward UI widget may lead to submitting all (1-65535) ports instead of selected ones.

Action Points

  • Be more conscious adjusting NAT settings in OPNsense ¯\_(ツ)_/¯

  • Ensure that Flareon is accessible using Hetzner Cloud Console

  • Introduce more resilient way to access internal network without relying on firewall settings.

    • Dedicated hardware device

      • Add RTC to make sure expiry-date keys will work after power cycle

    • Ethernet connection to Internal network

    • Back-connected WireGuard tunnel to external node

    • NAT port-forwarded port

    • Backup cellular connection?

    • Another obscure connectivity? r-node? RTTY?