On a Friday afternoon two of our hypervisor nodes dropped off the LAN at the same moment, along with the inference VM on one of them and several other hosts. From the gateway, ARP for each of them came back incomplete. From the one hypervisor node that was still reachable, the same: FAILED. No ping, no SSH, no management UI, nothing at layer two. For about an hour and twenty minutes, most of the estate did not exist as far as the network was concerned.
The larger of the two nodes had been behaving badly for two days before that — leaving and rejoining its corosync cluster, dropping quorum, coming back. So when it finally disappeared for good, the story wrote itself: the node had crashed, the cluster was down, and the memory on that machine had a known history. The story was wrong. The node never crashed. A switch did, and every piece of evidence that pointed at the server was an artifact of looking at it from the wrong side of that switch.
Key Takeaways
- When several hosts fail at layer two together — ARP incomplete or failed, not merely unreachable at a port — while the gateway and a host on a different switch stay up, the shared switch is the first suspect, not the servers behind it
- A node that leaves and rejoins its cluster for days before vanishing is not necessarily a node that is failing; it is just as consistent with a switch that is failing in front of it, and only the switch's own state distinguishes the two
- A Proxmox node's journal will show nothing but
no quorumthrough a network partition it cannot see past, and a clean shutdown entry at the end is positive evidence that the server itself was healthy - A memory-error daemon reporting zero errors during the window rules out the hardware story the node's history invited; check it before accepting a crash narrative
- A centralized log collector that runs on a guest behind the same switch receives nothing from the outage window, and the gap looks like "nothing happened" rather than "nothing arrived"
- The network controller's own alert database on the gateway is the ground truth for switch state, and it is worth knowing how to query it directly rather than through the dashboard
- After the node returns, a VM marked to start on boot can stay stopped because autostart waits on quorum; check guest state by hand rather than trusting the boot flag
What It Looked Like From the Outside
The gateway, a UniFi device, was fine. It could see its own uplink and could see the one hypervisor node that sits on a different switch. From that surviving node, every other hypervisor host and every guest behind them was unreachable, and the ARP table said why in the only vocabulary it has: INCOMPLETE, then FAILED. Not a closed port, not a host rejecting connections, not a service down. Nobody was answering the question "who has this address" on the wire.
Several other hosts went with them — a documentation server, a management appliance, a development hypervisor. The list was not random. It was every device plugged into one switch.
That pattern is the entire diagnosis, and we did not read it that way at first.
The Two-Day Prelude That Pointed the Wrong Way
The larger node had been unstable for two days. Its corosync membership kept flapping — it would lose contact with the other nodes, the cluster would lose quorum, the node would rejoin, and the cycle would repeat. We had looked at it, seen a node that could not hold a cluster connection, and provisionally blamed the node. This particular machine has a documented memory history, so a hardware story was already on the table.
That read was consistent with the evidence. It was also consistent with a switch in front of the node intermittently dropping frames for two days before failing completely. Corosync does not know the difference. From inside the node, a link that drops cluster traffic for a few seconds every so often is indistinguishable from a NIC that does, or a CPU that stalls. The node's journal recorded what the node experienced — no quorum, repeatedly — and the node experienced a network it could not see past.
The thing that should have redirected us earlier was that the other hypervisor on that switch was also flapping its cluster membership in the same window. Two nodes with independent hardware do not develop the same intermittent fault on the same two days. One switch does.
The Instinct We Did Not Follow Through On
The instinct, with two hypervisors dark and an inference VM unreachable, was to go to the rack and power-cycle. It is the instinct for a reason: it resolves a wedged server, it is fast, and it feels like doing something.
It would have resolved nothing. The servers were running. Power-cycling them would have cost a dirty shutdown of every guest, a filesystem check on the way back, several minutes of cluster rejoin, and a false sense that the reboot had fixed something — because the switch came back on its own eighty minutes later, and a reboot timed anywhere near that would have taken the credit.
The one thing that was actually done at the server was a clean reboot of the larger node after the switch had already recovered, which is what produced the journal's clean shutdown entry. That entry is worth more than it looks: a server that had crashed would not have one.
Where the Ground Truth Was
The switch is a UniFi aggregation unit that carries the uplinks for the hypervisor cabinet. The controller — which runs on the gateway — logs device events to its own database. Querying that database directly, on the gateway, gave the timeline nothing else could:
DEVICE_UNREACHABLEfor the aggregation switch at 16:30 local timeDEVICE_RECONNECTEDfor the same switch at 17:48
The larger hypervisor had last been seen at 16:26, four minutes before the controller declared the switch unreachable. That is not the node failing first; it is the switch degrading past the point of carrying traffic a few minutes before the controller's own keepalive gave up on it.
The query itself is unglamorous — a shell on the gateway and a read against the controller's alert collection, filtered by event type and time. Two things recommend it over the dashboard. It is scriptable, which the dashboard's event view is not. And it works when the usual API path does not, which was our situation: the controller's API credentials in our vault were stale and the key-based route returned authorization errors. Knowing the local query path meant the stale credential was an inconvenience rather than a blocker, on an afternoon when it would have been a blocker.
The Logging Blind Spot
We run centralized logging. Syslog from the network gear, including the switches and the gateway, ships to a Loki stack fronted by Vector, and under normal conditions the switch's own view of a link flap is in there within seconds.
For this outage window, there was nothing. Not an error, not a gap marker — an absence. The log collector runs as a guest on the larger hypervisor, which sits behind the aggregation switch that failed. When the switch went down, the gateway's syslog forwarder could not reach the collector. Every event the gateway and the surviving switches generated about the outage was emitted into a network that could not deliver it, and syslog over UDP does not retry.
That gap reads as "nothing happened" to anyone querying the logs afterward. It is not nothing; it is the absence of the specific messages that would have named the cause, during exactly the window that mattered. The collector was inside its own failure domain, the same shape as a monitor that cannot report a WAN outage because it reports over the WAN.
Two consequences. First, the controller's database on the gateway was the only record that survived, because the gateway is the one device that was never on the wrong side of the failed switch. Second, a log collector's placement is a design decision that gets made by default — "put it on the big node, it has the disk" — and the default places it where it is least useful for the outages that matter most. Moving it, or adding a second receiver on a host outside the hypervisor cabinet, is the follow-up.
The Rule
When several hosts fail at the hardware level at once, and the set of failed hosts is the set of hosts behind one switch, and the gateway and anything on another switch stay up: check the switch first. Look at the controller's view of the switch, not the servers' view of the network. A server power-cycle will not help, and it will cost you a clean record of what the servers actually experienced.
The pre-conditions are each necessary. One host down is a host problem. Several hosts down on different switches is a gateway or upstream problem. Several hosts down on one switch, with ARP failing rather than services failing, is that switch until proven otherwise.
The hardest part of applying the rule is that the servers behind the switch produce evidence that looks like server evidence. Their journals fill with cluster errors. Their monitoring goes dark in a way that resembles a crash. If one of them has a hardware history, that history will be the first thing anyone remembers. The memory-error daemon on our larger node reported zero errors for the entire window, and we checked it after the switch had already been identified. It should have been checked before the crash story was allowed to form.
Aftermath: The VM That Did Not Come Back
After the switch recovered and the larger node had been cleanly rebooted, the cluster reformed and most guests returned. The inference VM did not, despite being configured to start on boot.
Proxmox's autostart on a cluster node waits for quorum before starting guests. If the node comes up before quorum is established — which is easy in the minutes after a partition, while the other nodes are still sorting themselves out — autostart can decline to run, and a guest marked onboot stays stopped with no error that announces itself. The fix is to start it by hand once quorum is confirmed. The lesson is to check guest state after any cluster-wide event rather than trusting the boot flag, because the flag describes an intent, and the cluster has veto.
Honest Limits
This is a diagnosis after the fact. We did not have a switch-level alert that would have paged on DEVICE_UNREACHABLE for the aggregation switch, and we are not going to claim the controller's database query is a monitoring strategy. It is a forensic one. The monitoring gap — an alert on a core switch going unreachable, delivered by a path that does not pass through that switch — is the open item, and it is not closed.
We do not know why the switch went down. The controller recorded that it became unreachable and that it reconnected eighty minutes later. Whether it rebooted, lost power, or hung is not in any record we have, because the records that would say so were being sent through it.
The two-day prelude is the part we are least able to prove. It is consistent with a degrading switch. It is also consistent with something else on that switch misbehaving and the switch's final failure being coincidental. The ARP-level evidence settles the afternoon; it does not settle the two days before it.
Finally, the rule is a prior, not a proof. A switch failing and a server failing at the same time is unlikely and not impossible. "Check the switch first" means first, not only.
Where This Fits
A network partition produces server-shaped evidence on every server inside it, and a logging pipeline that lives inside the partition produces silence. Both of those are things a monitoring design has to be told about explicitly, because the defaults — collect logs where the disk is, trust the journals, believe the hardware history — all point at the wrong cause. We have written before about the gap between collecting security telemetry and being able to act on it, and about standing up a SIEM on a small estate; this incident is the infrastructure-side version of the same problem, where the collector's placement decided what the SIEM could ever have seen.
Deciding where the collectors live, which switches get their own out-of-band alert, and what a cluster does on the way back from a partition is managed infrastructure work, and the day-to-day of responding to an afternoon like this one is IT support.
If your hypervisors sit behind one aggregation switch and your log collector sits behind the same one, book a call and we will go through what you would and would not be able to see.