Every hour, for 11 days, a scheduled job on our inference host regenerated a set of inventory documents, committed them, tried to push, failed, discarded the commit, and wrote one line to its own log: push failed, self-corrects on the next run. Nothing reached the remote in that time. Nobody noticed, because the gate that blocked the push was working exactly as designed and nothing was built to care that it had fired.
What changed on day zero was that we installed gitleaks on that host. Until then the pre-push gate's secret-scan step could not run, returned exit 3 — UNKNOWN — and the gate warned and let the push through. With gitleaks installed, the scan ran, found that three tracked files carried live credentials — a tunnel credential and an API key — and returned exit 1. FAIL. Block. The gate was right. Everything around it was wrong, and this post is about the everything.
Key Takeaways
- A mechanical gate needs three outcomes, not two: 0 PASS, 1 FAIL, 3 UNKNOWN — "could not evaluate" is a different fact from "evaluated and passed" and must never be collapsed into either
- UNKNOWN that warns and passes is a reasonable interactive default and a dangerous unattended one; for a cron, "the scanner could not run" means every push goes through unscanned, on schedule, forever
- Installing a scanner can flip a gate from UNKNOWN to FAIL on the spot; the secrets were already there, and the tool's arrival is only what made the gate able to see them
- "Self-corrects" in a log is a claim, not a mechanism — a job that discards its work and retries from scratch is not converging on anything, and its log should say what it will take to unblock
- A blocked push is a silent failure unless something consumes the block; the remote stopped receiving commits and no alert existed for "expected commits stopped arriving"
- A secret scanner's zero is UNKNOWN, not PASS — it reports that no known pattern matched; its one is a real finding, and that asymmetry is the whole reason exit 3 exists
- Redaction removes a value from the working tree, not from history; every clone, mirror, and backup still carries it, so rotation is the fix and redaction is tidiness
Why Nothing Reached the Remote
The job itself is unremarkable. It enumerates the estate — hosts, containers, services, versions — and writes the result to generated markdown files in our infrastructure repository, so the documentation stays current without anyone maintaining it by hand. It runs hourly from cron, commits with a fixed chore(inventory) prefix, and pushes to the remote.
The push goes through a global git pre-push hook, the same one every repository on every machine of ours uses. The hook knows nothing about the repository; it calls the repository's own check script — the one answer to "is this safe to ship" — which in turn calls a shared secret-scan step, and the hook acts on the exit code.
On that host, until late August, the secret-scan step returned 3 on every run because gitleaks was not installed there. Exit 3 is UNKNOWN in our contract: the hook wrote a warning to a log file and allowed the push. The inventory went out every hour, warning attached, and nobody read the warning because the pushes were succeeding.
Then gitleaks was installed, as ordinary tooling hygiene, with no intention of changing anything about this job. On the next hourly run the scan executed for the first time. Three of the generated files contained real credential values captured from configuration during enumeration — a tunnel credential and an API key, rendered straight into the documentation. gitleaks matched them. The step returned 1. The hook blocked.
From that run forward the job's behaviour was: generate, commit, push, get blocked, reset to discard the commit, log "push failed … self-corrects", wait an hour, repeat. Eleven days.
The Gate Did Exactly What It Was Built To Do
Be precise about what failed here, because the obvious reading — "the gate broke the cron" — is wrong, and acting on it produces the wrong fix.
The gate found live credentials in files about to be pushed and refused the push. That is the only correct behaviour. Those three files had been tracked, committed, and pushed hourly for as long as the job had run with the scanner absent, so the remote already had the values in its history. The gate did not cause that. It was the first thing in the chain to notice it, on the first run it was able to.
What failed was everything that should have surrounded the gate:
- Nothing told a person the gate had moved from UNKNOWN to FAIL.
- Nothing told a person that a job which had pushed every hour had stopped pushing.
- The job's own log described the block as a transient condition that would clear itself.
- The job's recovery logic — discard and retry — was incapable of clearing it.
A control that blocks correctly and is then ignored for 11 days is not a control failure. It is an observability failure wrapped around a working control, and the fix is to build the observation, not to loosen the gate.
0, 1, 3: Three Outcomes, Not Two
The contract our check scripts follow fits in a comment, and it is the part of this incident most worth stealing:
# exit-code contract for every mechanical gate
# 0 PASS evaluated, nothing found
# 1 FAIL evaluated, finding — block
# 3 UNKNOWN could not evaluate — never a silent 0
Most gates have two outcomes, 0 and non-zero, and that is the problem. When a scanner is missing, a temp directory is unwritable, or a tool crashes before producing a verdict, a two-outcome gate has to pick. Pick 0, and a broken gate passes everything silently, because a gate that cannot run and a gate that ran clean look identical. Pick 1, and every tooling hiccup blocks every push until the gate gets disabled.
Exit 3 is the third answer: I do not know, and here is why. The caller decides what to do with that. It is a strictly more honest contract, and it is why this is a story about an 11-day gap rather than about indefinitely pushing credentials through a gate that had never scanned anything.
A second instance on this estate illustrates the other edge. On one workstation the check script runs inside a sandbox that denies it a temporary file, so the scan returns 3 on every push and the hook warns and passes. That is the contract working as written — and it means that machine has never once scanned a push. Which brings us to the real design question.
UNKNOWN Must Not Pass Unattended
The gate's rule — exit 3 warns and passes — was chosen for a person at a keyboard. A developer sees the warning, knows the scanner is missing or broken, and makes a judgment. That is defensible: the one person who can act on the warning reads it as it is produced.
A cron job reads nothing. When the unattended inventory push received a 3, the warning went into a file and the push went through — unscanned, on schedule, with a log line that said so to nobody. For an unattended caller, UNKNOWN-passes is no gate at all, plus a false sense of coverage.
The contract needs one more line, and it belongs to the caller: an unattended caller treats 3 as a block. If the scanner cannot run, the cron does not push. It fails loudly, which is the only way an unattended process can ask for help. The developer keeps warn-and-pass; the job nobody is watching does not get the benefit of the doubt, because there is nobody to extend it.
This is also where the scanner's own output meets the contract, and where the contract has to stay machine-readable. A secret scanner's zero is not a clean history — a zero means no known pattern matched, which is weaker than absence, and we have three confirmed misses on this estate to prove it. The temptation is to express that humility in the exit code and never return 0 from a secret scan. Do not: an unattended caller that blocks on 3 would then block every clean push forever. The split is strict. 0 means the scanner ran to completion and matched nothing. 3 means the scanner did not evaluate — not installed, no temp directory, crashed, timed out. The humility lives in the detail line, which for a clean run reads "no known-pattern match (not proof of absence)" rather than "clean", so a human reading the log is not misled while the cron, which only reads the code, passes on 0 and blocks on 3. The exit code carries the category. The message carries the reason.
"Self-Corrects" Was a Claim, Not a Mechanism
The single line that cost the most was in the cron's own log:
push failed — self-corrects on next run
That line was written by us at a time when the only push failures the job had seen were transient: a network blip, a remote briefly unreachable, a lock held by another process. For those, discard-and-retry does self-correct. The next hour's run regenerates, commits, pushes, succeeds.
A gate block is not transient. The condition that produced it — three tracked files with credential values — is regenerated identically on every run. Discard-and-retry against a deterministic block is not recovery; it is the same failure with a timestamp. The job was not converging. It was looping, and its log said it was healing.
A log line that promises self-correction should be allowed only where the code can demonstrate the mechanism. If the retry logic is "try again later", the log should say "will retry", and it should carry the gate's exit code and detail line verbatim so a reader can tell a transient from a block. "Push blocked: secret-scan FAIL, findings in 3 tracked files" would have been read very differently from "self-corrects".
Nobody Was Watching for the Absence
The detection, when it came, was not an alert. It was someone noticing that the repository's main branch had not received an inventory commit in a while, and going to look. The job's log on the host — now documented as the first place to read — had the answer on its first screen.
The gap is a familiar one. We monitor for things that happen. The inventory job did not error in any way cron notices — it swallowed the push failure, reset, and exited 0 — and the remote simply received nothing. There is no event in "nothing arrived", and a monitoring stack built on events does not see it.
The check that catches this is a dead-man's switch: the job pings a monitor on a successful push, and the monitor alerts when the ping is late. We run that pattern for other scheduled work and had not extended it to this job because its output felt low-stakes. The output was low-stakes. The silence was not — eleven days in which the one control that had finally noticed credentials in a remote-bound repository was being quietly overridden by a reset.
Every unattended push needs two things: a block on UNKNOWN, and an absence alert on the remote. The first makes the job fail honestly. The second makes someone hear it.
Redaction Is Not Rotation
Once the cause was clear, the fix to the files was the same day: the three offending documents were redacted and committed, and the gate had nothing left to match.
That is where a lot of incident write-ups stop, and it is the wrong place. Redacting a file changes the working tree and the newest commit. It does nothing to the history that already contains the values — on the remote, in every clone, in the mirror on our NAS, in backups of all of those. The values are as retrievable as they were before the redaction, by anyone with a copy of the repository and git log -S.
Rotation is the fix. Redaction is tidiness. A tunnel credential and an API key that have been in a repository's history have to be treated as disclosed, and the only remediation that closes a disclosure is making the value worthless. We recorded rotation as the remaining step rather than treating the clean push as closure, and we would say the same to any client: history rewriting helps and does not substitute for invalidating the credential.
A generator that renders a token into a markdown file is an upstream design fault, and a scanner catching it at push time is the last line, not the first. The fix that prevents recurrence is a generator that never sees credential values, or redacts at the source and fails closed. That is the next change on this job.
Honest Limits
This is one job, one host, one gate contract, and the details — hourly cadence, generated markdown, a global pre-push hook — are ours. The contract generalizes; the shape of the cron does not. If your unattended pushes go through CI rather than a local hook, the same question applies to the pipeline's handling of a scanner that is absent or crashes: does it fail, pass, or distinguish.
The 0/1/3 contract is not a standard. It is a convention we chose because it fits in a shell exit code and 2 is already taken by enough tools to be ambiguous. Any three-way encoding works; what matters is that "could not evaluate" has its own value. The absence alert likewise assumes a known cadence — a job that pushes only when there are changes needs a maximum-silence window instead, which is a judgment rather than a formula.
And none of this makes the scanner better at finding secrets. gitleaks found these three files because the values matched patterns it ships; the classes of credential it cannot see are documented in the earlier post. A gate that blocks correctly on a finding still passes silently on a miss, and the only defence against the miss is the content-search habit — searching history for the specific value at risk — that this incident was lucky enough not to need.
Where This Fits
This post builds on what a secret scanner's zero does and does not prove. That one is about the scanner's output. This one is about what happens when that output is correct and the system around it has no way to hear it — the three-way contract, the interactive versus unattended caller, and the log line that turned a deterministic block into eleven days of imagined recovery.
The gate design and the absence monitoring are DevOps engineering work. The credential handling — a generator that must never render a token, the rotation that follows any disclosure, the language that distinguishes "no known-pattern secrets" from "clean" — is cybersecurity, specifically the security infrastructure that turns a scanner into a control.
If your pre-push or CI gates have a pass path for "the scanner could not run", book a call and we will go through which of your unattended jobs have been shipping through it.