Entry 06Platform · reliability2026

When your edge is green but your service is dead

Cloudflare Tunnel can report a healthy edge while the service behind it is silently broken. I built an autonomous sentinel that watches the origin, not the edge — and deliberately shipped it without write access until it earned trust.

watchesthe origin
edge reportshealthy
shipped withread-only
remediationstaged

The failure nobody monitors for

Edge and tunnel status answer one question: is the tunnel connected? They do not answer the only question that matters — is the thing behind the tunnel actually alive?

A container reboots, DHCP hands it a new LAN IP, and the tunnel ingress still points at the old one. Now the edge serves errors — or worse, a different service — while every dashboard stays green. I call this the silent-origin-behind-healthy-edge class. Standard monitoring is structurally blind to it, because it probes the wrong layer.

Classify before you heal

The sentinel probes each origin LAN-side, bypassing Access, and classifies the real state before it touches anything. Only when a container's IP has genuinely drifted does it propose a Cloudflare ingress repoint.

Everything it learns is written to a machine-readable health.json, so the next automated agent — or a human — inherits ground truth instead of re-diagnosing from scratch.

origin state
OK
STALE_INGRESS_IP   -> propose repoint
DEAD_ORIGIN
TUNNEL_DOWN
APP_UNHEALTHY
NEEDS_HUMAN

The part that matters: restraint

Auto-remediation that can rewrite production ingress is a loaded gun. The engineering here is mostly about not firing it:

  • Report-only v1. It proposes repoints into health.json and writes zero Cloudflare changes. Write access is a later, earned phase — not day one.
  • Read-scoped token. It cannot mutate what it isn't authorized to.
  • Blast-radius guards. Localhost origins are never auto-repointed; heal confirmation requires a stricter 2xx; a repoint preserves every existing config key and handles multi-rule hostnames; one origin's failure never aborts the run; heal-state writes are atomic.
  • Testable by construction. A pure functional core — data in, data out, no I/O — with all side effects pushed to the shell edge, and 63 tests against the part where correctness is non-negotiable.

What this demonstrates

Two things I bring to infrastructure work: I find the failure classes your current monitoring can't see, and I build remediation that defaults to do no harm — staged rollouts, least-privilege tokens, isolated failure domains, and an architecture where the risky logic is the tested logic.

If you run agents, tunnels, or self-healing systems in production, that combination is the difference between automation you trust at 3am and automation you switch off after it burns you once.

Platform / SRE Cloudflare Python Self-Heal

Interested in this kind of work? Email is best — [email protected] · Back to portfolio