The health endpoint answered 200 in eleven milliseconds. systemctl is-active said active. Every monitor was green, and had been green throughout. Meanwhile a generation request that normally returned in seconds hung past three minutes, because an unattended package upgrade had left the loaded NVIDIA kernel module behind the installed userspace, and the inference server had quietly fallen back to running a 14-billion-parameter model on the CPU.
Nothing errored. That is the entire problem, and it is the recurring shape of self-hosted inference: a hosted API fails loudly with a status code you can alert on, while your own box degrades into a state that is slower by orders of magnitude and indistinguishable from healthy to every check you thought to write. What follows are three of those states we have hit on our own inference host, what each looks like from the outside, and the checks that actually separate them.
Key Takeaways
"the API answers"is not"the model is on the GPU"— a liveness endpoint stays green through a complete GPU outage, because listing available models never touches the accelerator- A driver/library version mismatch does not stop the service; it makes it fall back to CPU and keep serving correct answers at roughly the wrong order of magnitude
ollama psand itsPROCESSORcolumn is the fastest placement check, and it is also the check that lies during the second failure mode — no single probe covers both- Purging an obsolete driver package flags the entire current driver stack as auto-removable, so a later routine
apt autoremovedeletes your working GPU driver - A wedged scheduler queue returns instant
503s on every endpoint while the model sits loaded at 100% GPU, and is usually caused by an unthrottled client, not by load - Disabling a reasoning model's thinking makes it fast and makes it skip tool calls — it narrates the action instead of taking it
- Health checks for inference must assert placement, not liveness: report GPU offload percentage from the process endpoint and fail the check when it drops
Failure One: The Driver Upgrade That Kept Serving
An unattended nvidia-driver package upgrade installed a new userspace while the previously loaded kernel module stayed resident. The result is a version skew that only announces itself if you ask directly:
Failed to initialize NVML: Driver/library version mismatch
NVRM (loaded): 580.159.03 # from before the upgrade
userspace: 580.173.02 # what apt installed
Two point releases apart. nvidia-smi reports the NVML error immediately, which sounds like it should be obvious — but nothing runs nvidia-smi on a schedule, and nothing downstream was written to care about it.
Here is what the layers above actually showed:
systemctl is-active ollama→active/api/tags→200in 11ms, so every naive health check passed/api/generate→ hung past 180 secondsollama ps→PROCESSOR = 100% CPU
That fourth line is the whole diagnosis, and the first three are why nobody looked at it for a while. A 14B model with a 40k context window on CPU takes minutes per call rather than seconds. It still produces correct output. It is not broken in any sense a monitor recognizes — it is simply running somewhere else.
The failure surfaces downstream as something that looks unrelated. What we actually noticed was an internal dashboard panel that had rendered instantly for months beginning to spin for thirty seconds and sometimes time out. Nothing in that symptom points at a GPU driver. The path from "a panel is slow" to "a kernel module is two point releases behind userspace" is several unintuitive hops, and every hop is a place to conclude the application is at fault.
Diagnosing it directly. Compare the loaded module against the installed package — cat /proc/driver/nvidia/version versus dpkg -l | grep nvidia-driver. Check placement with ollama ps and read the PROCESSOR column. Confirm what the engine chose at load time with journalctl -u ollama | grep "inference compute", which on a healthy host names the CUDA library and the specific card.
Fixing it. The fix is a reboot, to load the module matching the installed userspace. One caution that will waste your evening if you skip it: confirm DKMS has already built the module for the running kernel first, with dkms status and modinfo nvidia | grep version. If it has not, the reboot changes nothing and you have spent a maintenance window learning that. Budget for the reload as well — a 14B model takes roughly 210 seconds to come back onto the GPU after the host returns, during which the box is up and the service is still slow.
The Footgun You Arm While Cleaning Up
Having fixed the mismatch, the tidy instinct is to purge the obsolete driver package. That instinct is the second incident.
Simulating it first is what caught it. apt-get -s purge nvidia-driver-550 removes only that metapackage — but it marks the entire current stack, the driver, the DKMS package, and every libnvidia-* component, as "automatically installed and no longer required." Nothing breaks at that moment. The damage is scheduled: the next routine apt autoremove, run by a person tidying disk space or by whatever cleanup you have automated, deletes the working GPU driver.
The guard is one command, before the purge rather than after:
sudo apt-mark manual nvidia-driver-580
sudo apt-get -s autoremove # verify nothing nvidia is listed
Simulate destructive package operations and read what they mark, not just what they remove. The dangerous part of that purge was never in its own output — it was in the state it left behind for a command somebody would run innocently three weeks later.
Failure Two: The Queue That Wedged While Everything Looked Healthy
The second mode is the inverse of the first, and it is why no single check is sufficient.
Over roughly two days, every inference call — chat, generate, embeddings, all of them — returned 503 "server busy, maximum pending requests exceeded". And ollama ps looked perfect: model loaded, 100% GPU, exactly what you want to see. The placement check that diagnosed the first failure reported health through the entire second one.
The recognizable signature is the timing. These were not slow requests eventually giving up; they were instant 503s, around 40 milliseconds, on every endpoint at once. A genuinely overloaded server is slow before it refuses. A server refusing in 40 milliseconds has a wedged queue, not a capacity problem.
The cause was a client, not the server. A scheduled indexing job was hitting the API at roughly 17 requests per second with no backoff, and its retry storm was plainly visible in the service journal. The scheduler's pending queue never drained.
The fix is unsatisfying but correct: kill the flooding client, then systemctl restart ollama. The part worth internalizing is what it means for everything that ran during the window — every result produced against a wedged queue is junk, not data. An automated security scan burned four hours against ours and produced a clean-looking report with no evaluation rows in it. That output was available to be quoted. It described nothing.
Two guards followed from it. The indexing client got exponential backoff and now aborts its run outright after consecutive failures rather than retrying into a wall. And a weekly audit re-pulls registry models and rescans configuration for models that are referenced but no longer present — a class of bug that had already cost us a monitor returning 404 every thirty seconds for weeks after a model was removed from the host but not from the config that called it.
Failure Three: Speed That Costs You Tool Calls
The third mode is not an outage at all, which is what makes it the most likely one to ship.
Chasing latency on a tool-using agent loop, disabling the reasoning model's thinking is the obvious lever. It works, spectacularly: sub-second responses where there had been tens of seconds. It also broke the agent, quietly. With thinking disabled, the model narrated its intended tool call — "I'll fetch that now" — without emitting a tool call at all, in three of four evaluation runs. The response is fluent, plausible, well-formed, and describes an action that never happened.
That failure is far more dangerous than a timeout. A timeout is an error you handle. A confident description of an action that was never taken flows straight into whatever consumes the output.
Reasoning models keep thinking ON for tool use. The latency was never coming from the think tokens, and the real cause is worth more than the lever we nearly pulled: it was residency. Each step was paying a cold load, and two models were fighting over the same 24 GB card, evicting each other in a seesaw. Sizing them to co-reside — an 8B at a 24576-token context using 7.0 GB, alongside a 14B at 40k using 12 GB — plus a long keep-alive and a warm-up at startup, took steps from 52–56 seconds down to roughly 13 seconds warm. Same models, same thinking, same hardware.
When local inference is slow, suspect residency before you start removing capability. Cold loads and VRAM eviction account for most of it, and both are configuration rather than a trade against quality.
Health Checks Must Assert Placement, Not Liveness
The single change with the most value across all three modes is to stop letting a liveness endpoint stand in for a functioning one.
Listing available models does not touch the GPU. It answers in milliseconds from a wedged queue, from a CPU fallback, from any state short of a dead process — which is precisely why it stayed green through an entire GPU outage while generation hung past three minutes. Any check built on it is measuring that the process is running, and nothing else.
The process endpoint knows better. Ollama's /api/ps reports what is loaded and how it is placed, and a health check can read GPU offload percentage from it and fail when the model is resident on the CPU. Ours now surfaces that as an explicit field and can be configured to require GPU placement outright, so a fallback is a failed health check rather than a mysteriously slow afternoon.
The general form applies well beyond inference: a health check should assert the property you actually depend on. Depending on GPU-speed generation and checking that a process answers HTTP is not a weak version of the right check. It is a different check, and it will pass in exactly the situation you built it to catch. That is the same failure shape as a secret scanner reporting zero findings — a control returning a confident green that carries no information about the thing you cared about.
Never Let an Optional LLM Step Block a Deterministic Answer
One architectural note, because the driver incident only became visible through it.
The slow panel was computing its data deterministically in 32 milliseconds and then calling a model to reword a short sentence — a cosmetic enhancement, on the blocking path, with the response gated on its completion. When the model went to CPU, every request paid the full timeout. Worse, failures were never cached, only successes, so there was no circuit breaker and no degradation: each request rediscovered the problem at full cost, forever.
Serve the deterministic answer immediately, produce the model's text in the background, cache it aggressively, and back off on failure so a broken model costs one slow request rather than all of them.
One subtlety worth stealing: check your cache TTL against your model keep-alive. Ours were both thirty minutes, so the cached text expired in lockstep with the model unloading, and every cache miss was guaranteed to also be a cold load. The cache had been scheduling its own worst case. Raising the TTL to 24 hours decoupled them.
Honest Limits
These are three modes we have hit and can describe precisely. They are not a complete taxonomy of how a self-hosted inference host degrades — thermal throttling, ECC memory errors, PCIe link degradation, and storage latency on model loads all produce their own quiet slowdowns, and we have not had to debug all of them on this hardware yet.
The specific numbers are ours. Load times, context sizes, and residency budgets are functions of a particular card, a particular set of models, and a particular storage path. The 24 GB co-residency arithmetic does not transfer to a 12 GB card, and the 210-second reload does not transfer to a smaller model. The methods transfer; the numbers do not.
The placement check assumes an engine that exposes placement. Ollama does. If you run inference behind a stack that reports only liveness, the equivalent check is a synthetic generation against a known-cheap prompt with a latency threshold — cruder, more expensive, and considerably better than checking that a port is open.
Finally, none of this argues that self-hosting is a mistake. It argues that self-hosting converts a class of loud vendor errors into a class of silent local degradation, and that the operational work is largely about converting them back.
Where This Fits
Running your own inference is a real cost saving and a real capability, and it is also an operations commitment that begins the day the box works. The failure modes above share one property: every one of them was invisible to monitoring that had been designed around whether services were running.
If you are weighing the trade in the first place, we wrote separately about the self-hosted versus cloud decision, and about building the inference hardware itself — this post is what owning it looks like after the build is finished.
Keeping GPU hosts, drivers, and the services on top of them honest is infrastructure work; designing the applications above them so an optional model call cannot take down a deterministic response is AI automation work.
If you are running local inference and your health checks confirm that the service is up rather than that the model is on the GPU, book a call and we will go through the placement checks with you.