We stopped the tunnel connector on the primary hypervisor node, on purpose, to prove that the standby would take over. It did not. The standby's monitor thread was polling, the peer's health endpoint was answering, and from the standby's point of view nothing had gone wrong — because the thing it was watching was the small webhook server that sits next to the connector, not the connector itself. The connector was down. The watcher was up. The standby was satisfied.
That was the first of four ways this setup produced a confidently wrong answer. The design is simple to describe — one Cloudflare Tunnel, two hypervisor nodes, one connector running at a time — and every part of it that failed did so quietly. This is the full shape of what we built, the timings we measured when it finally worked, and each trap in the order we fell into it.
Key Takeaways
- A standby must watch the thing that serves traffic, not the agent that reports on it — a monitor polling a sidecar's health endpoint will never see the connector it is supposed to replace go down
- Failover and fail-back are two separate mechanisms with two separate clocks; if you build one, test the other, because they race each other on a stop-only test
- A fail-back timer needs a reachability check that does not depend on your own tunnel, and it needs two consecutive green runs, not one, or a flapping link hands the tunnel back and forth
- A maintenance flag file is the cheapest possible control for "do not fail back yet" — a standby serving while you work on the primary is only useful if the primary stays down until you say so
- Small Python socket servers that restart under systemd must set
SO_REUSEADDR, or every restart crash-loops onEADDRINUSEfor as long as the old socket lingers in TIME_WAIT, with the monitor dead the whole time - The primary must check the peer's connector readiness endpoint, never the peer's own status endpoint — two watchers each asking the other "how are you" recurse into a stall that reads as a failure
- Silence the alert rule for the duration of a gate test; the alert is doing its job and will page on a planned stop within a minute
One Tunnel, Two Nodes, One Connector at a Time
The ingress for our public services is a single Cloudflare Tunnel. The connector, cloudflared, runs on the primary hypervisor node. A second node runs an identical connector configuration for the same tunnel, stopped. Next to the connector on each node is a small failover server — a short Python service under systemd — that exposes a health endpoint and a token-protected webhook for starting and stopping the local connector.
The standby's copy of that server has one extra piece: a monitor thread that wakes every 30 seconds, checks the peer, and decides whether to take over. The primary's copy, since early September, has a fail-back timer that decides whether to take back. Everything in this post is about those two decisions.
If you run multiple connectors concurrently as replicas, the fail-back half of this does not apply to you. The monitor traps still do, because the question "is the other node actually serving" has to be answered by something, and that something can be wrong.
The Standby Watches Two Signals, Three Times
The monitor's rule after the fix is deliberately conservative. On each 30-second tick it checks two things about the peer: whether the peer's health endpoint answers, and whether the peer's connector is ready — not merely whether its process exists. cloudflared exposes a /ready endpoint on its metrics port that returns 200 only while the connector holds at least one live connection to Cloudflare's edge, so it answers the question the monitor actually needs answered: is this node serving traffic right now. The peer counts as healthy only when both are good; a miss is recorded the moment either one is bad. Three consecutive misses trigger a takeover:
every 30 s:
peer_health = GET peer /health -> 200 or not
peer_ready = GET peer cloudflared /ready -> 200 only with a live edge connection
miss = not (peer_health and peer_ready)
misses = misses + 1 if miss else 0
if misses >= 3:
systemctl start cloudflared # serve locally
POST peer /failover/stop # tell the peer to stand down, if reachable
When the peer's connector reports ready again, the standby stops its own and returns to standby. Three misses at 30-second spacing means a takeover lands somewhere between 60 and 90 seconds after the real failure, depending on where in the poll cycle the failure falls. That is slow enough to ride through a single dropped poll and fast enough that nobody has to be paged to do it by hand.
Our first version of the fix used a weaker second signal — "the peer reports its cloudflared process as running" — and it is worth saying why that is not enough. A process can be running with no connection behind it: the primary loses its route to Cloudflare while staying reachable on the LAN, systemd still reports the unit active, the health server still answers, and both checks pass while the tunnel has no connector. Process state catches a stopped connector; readiness catches a stopped and an isolated one. Poll the connector's own readiness, not a bystander's opinion of its process table.
The "and" in that rule — healthy means health and ready, never one without the other — is the whole lesson of the first trap.
The Test That Never Failed Over
Before the fix, the monitor checked only the peer's health endpoint. That endpoint belongs to the failover server, not to the connector. Stopping cloudflared on the primary leaves the failover server running and healthy, so the standby saw a green peer, recorded zero misses, and did nothing. The tunnel had no connector. Every public service was dark. The monitor whose only job was to notice this was reporting that everything was fine.
The monitor was not malfunctioning. It was checking exactly what it had been written to check, and that thing was healthy. The wrong turn is in the design, not the runtime: the obvious thing to poll on the peer is the endpoint you wrote, and the endpoint you wrote is the one component on the peer whose state has nothing to do with whether traffic is being served.
The fix is to make the connector's own readiness part of the health decision, which is what the two-signal rule above does. The general form is worth stating plainly, because it is the same mistake as a liveness check standing in for a functioning service: a health endpoint reports the health of whatever answers it. If the thing that answers is not the thing that serves, the check is measuring a bystander.
Fail-Back Is a Timer, and a Flag Holds It Down
Failover without fail-back leaves the standby serving indefinitely, with the primary's connector stopped and nothing to restart it. Before September 8 that was our position: a takeover was a one-way trip until someone noticed.
The fail-back mechanism is a systemd timer on the primary, firing every minute, running a short script. The script's logic:
# runs every 60 s on the primary
systemctl is-active --quiet cloudflared && exit 0 # already serving, nothing to do
[ -e "$MAINTENANCE_FLAG" ] && exit 0 # held down on purpose
if curl -fsS --max-time 5 https://www.cloudflare.com/cdn-cgi/trace >/dev/null; then
count=$(( $(cat "$STATE" 2>/dev/null || echo 0) + 1 ))
else
count=0
fi
echo "$count" > "$STATE"
if [ "$count" -ge 2 ]; then
systemctl start cloudflared
notify "fail-back: primary connector started" # a log-class note, not a page
echo 0 > "$STATE"
fi
Three properties matter. The reachability check goes to Cloudflare's trace endpoint directly, not through the tunnel, so it answers the question "does this node have egress to Cloudflare" rather than "is the tunnel up", which is what the fail-back is about to fix. It requires two consecutive green runs, so a link that is flapping does not hand the tunnel back and forth every minute. And it is blocked entirely by the presence of a flag file.
The flag is the part we use most. Touching it on the primary holds the primary's connector down regardless of what the timer sees, and the standby serves meanwhile. Removing it hands the tunnel back within about two minutes. That is the whole maintenance procedure for the primary node: touch, work, remove. Nothing to remember about the standby, nothing to restart by hand.
What the Gate Measured
With both halves in place we ran a gate test on September 8, with the flag set so that fail-back could not interfere with the failover half. Measured from the moment the primary's connector was stopped:
- +70 s — standby serving. Three misses at 30-second spacing, landing on the early side of the 60–90 second window.
- flag released, +102 s — primary connector back. Two consecutive one-minute timer runs after the flag disappeared, plus connector startup.
- +18 s after that — standby stood down, having seen the peer report its connector running on the next monitor tick.
Those numbers are the ones we now hold the design to. A change to the poll interval, the miss count, the timer period, or the consecutive-run threshold changes them, and the gate is re-run rather than reasoned about.
One operational note for anyone repeating this: silence the alert rule first. Our Prometheus rule for the primary connector fires after one minute down, and a gate test spends a good deal longer than that in states the alert is correctly designed to catch. We silence alertname=~"Cloudflare.*" for the test window and remove the silence afterward. An alert that pages during a planned test is not a false positive; it is the reason you would want it on an unplanned day.
A Stop-Only Test Makes Fail-Back Beat Failover
The second trap appeared the first time we tested without the flag.
Stop the connector on a primary that still has network egress, and two clocks start. The standby's monitor needs three misses, roughly 90 seconds. The primary's fail-back timer needs two consecutive green runs, roughly 80 seconds — and the primary has egress, because nothing is actually wrong with it, so every run is green. Fail-back wins. The primary restarts its own connector before the standby ever reaches three misses, and the tunnel simply has no connector at all for about 80 seconds.
The wrong reading of that result is "fail-back is too aggressive" or "failover is too slow". The right reading is that the test does not resemble the failure it is supposed to simulate. A real isolation of the primary — the node loses power, loses its uplink, loses its route to Cloudflare — has no egress, so the trace check fails, the count stays at zero, and the standby takes over unopposed. A stop-only test models a process crash on a healthy node, and for that case a fast local restart is exactly the correct response.
So the result is expected, and the rule that follows from it is: to test the failover path, remove the primary's egress or set the flag; a bare stop tests fail-back. We write both into the runbook so that the next person running a gate does not spend an evening rediscovering which clock they were watching.
Two Ways the Watchers Themselves Failed
The remaining two traps were in the failover servers rather than in the logic they implement.
Crash-looping on a lingering socket. The servers are small Python socketserver instances. The original code did not set allow_reuse_address, which is Python's switch for SO_REUSEADDR on the listening socket. Under systemd, every restart of the service tried to bind the same port while the previous socket was still in TIME_WAIT, failed with EADDRINUSE, was restarted by systemd, and failed again — for about 50 seconds each time, until the kernel released the address. The service came back eventually, which is why it took a while to notice. But for those 50 seconds the standby's monitor thread did not exist, and a restart of the failover server was the most likely thing to happen during exactly the maintenance work where failover coverage matters. One line fixed it on both nodes:
class Server(socketserver.ThreadingTCPServer):
allow_reuse_address = True # SO_REUSEADDR is set before bind()
Recursing into a false failover. When we later gave the primary its own peer check, the first version asked the standby's failover server for its status. That status endpoint, in turn, reports on the peer — which is the primary, which is at that moment waiting for the standby's answer. Two watchers, each blocking on the other's view of the other, stalled long enough to register as a failed check and trigger a failover that nothing warranted. The fix is to make the primary check the standby's connector directly, through cloudflared's own readiness endpoint, which answers from the connector's state and knows nothing about peers. The rule generalizes: a peer check must terminate at the component being checked, not at another checker.
Neither of these produced an outage on its own. Both produced a monitoring layer that was less present than it appeared, and the first trap in this post was only found because we tested rather than trusted.
Honest Limits
This is a two-node design with a single tunnel and a single connector active at a time. It does not address a Cloudflare-side outage, which no amount of connector redundancy on our end can help; the trace check would fail on both nodes and both would correctly sit still. It also assumes the standby can reach the same origins the primary serves, which in our case it can; if it cannot, starting its connector only moves the failure.
The timings are ours and follow directly from the intervals we chose: 30-second polls, three misses, a one-minute timer, two consecutive runs. A different set of intervals gives a different gate, and the gate should be measured rather than derived. We also silence alerts manually for tests, which works at our scale and would not at a larger one.
Finally, the failover servers are hand-written and small. That is a feature for auditability and a liability for everything else. Both bugs in the last section are the kind that a more mature agent framework would have avoided by default, and the trade we made — understanding every line versus inheriting someone else's defaults — is one we would make again for this component and not for most others.
Where This Fits
Ingress is the one piece of self-hosted infrastructure where a single point of failure is also a single point of visibility: when the tunnel is down, every check that goes through the tunnel is down with it, and the only honest signals are the ones that reach Cloudflare some other way. That is why the fail-back check talks to Cloudflare directly and why the monitor has to look at the connector rather than at its neighbour.
If you are weighing whether to run this kind of estate yourself, we wrote about the self-hosted versus cloud decision separately; this post is the operational tax that comes due after that decision. Keeping tunnel connectors, failover logic, and the alert rules around them honest is managed infrastructure work, and the gate-test discipline — measure the timings, write the two clocks into the runbook, silence the rule on purpose — is the part of DevOps that outlives whichever tool is doing the serving.
If your public ingress has a standby that has never been proven to take over, book a call and we will run the gate with you.