Traders understand something about operations that most software teams learn the hard way: at the end of the day, everything must reconcile. The back office has been running reconciliation loops since long before Kubernetes existed — comparing every trade against every confirmation, chasing the mismatches, refusing to sleep until book and reality agree. So when Kabir, who runs the platform desk at a fictional mid-size broking firm we will call Meridian, started reading about GitOps, the familiarity hit him immediately. The model, stripped to its studs: the team declares how each system ought to be, files that declaration in a store where every version survives untouched, and hands the fetching and the matching to software agents that never get tired. He recognised the shape at once. This is the story of what happened next — including the Friday when the discipline itself nearly deleted his settlement ledger, and the six ordered moves that brought it back.

Why a brokerage adopted GitOps at all

Meridian’s old arrangement would embarrass anyone who has ever sized a position. The build pipeline held production credentials and pushed changes into the platform whenever it pleased, which meant a compromise of that pipeline was functionally a compromise of the exchange connection. Reviews were informal and rarely written down. And the true configuration lived partly in servers nobody had audited in years. The GitOps pitch landed easily: declarations in a versioned store, agents pulling from inside, every production change arriving as a reviewed request. One detail from the security literature sealed it — the compliance team’s favourite line, in fact — which is that in this model the repository itself becomes the credential, since whoever can edit the watched path can, in effect, deploy to production. Meridian promptly guarded that path the way a risk desk guards a trading limit: two-person approval, scoped access, no exceptions for seniors. The first three months were everything the brochures promised. The fourth month was the Friday.

The Friday a merge dropped the settlement file

A routine change to the production configuration collided with another branch, and whoever resolved the conflict — tired, on a Friday afternoon, exactly as the rulebook predicts — settled it by deleting one side of the collision. The deleted side contained the settlement service’s manifest. Nobody noticed the merge land. The reconciler noticed, though, because noticing is its whole job: the live platform now contained something the declared state did not describe, and the agent did precisely what it had been configured to do all quarter. It pruned the settlement namespace. It also deleted the persistent volume behind it, because that volume’s reclaim policy said Delete — tidy, automatic, and catastrophic. The back-office equivalent is hard to overstate: the system of record for settlement instructions, gone, at 4:40 p.m., on the day margins are computed. The agent had not malfunctioned. It had obeyed the ledger, and the ledger had a hole in it.

The GitOps save: six moves in order

What follows is the recovery, reconstructed from the incident review, in the order the runbook prescribed:

1. Freeze the automation. Stop the reconciler from “helping” while humans think. Flux’s suspension survives cleanly because it is not itself declared in the store; on Argo CD the equivalent needs care, since parent applications will otherwise undo the pause unless their differences are explicitly ignored.
2. Assess the blast radius. Application history, cluster events, and the store’s log together answered the only question that mattered: what vanished, and what still exists?
3. Revert through the front door. A pull request restored the deleted manifest, reviewed like any other change — no heroic hand-edits, even now.
4. Restore the data. The volume was not coming back from the store; it came from backup, which is why backups exist and why “the repository is our backup” is a sentence that should never be said aloud.
5. Resume and verify. Automation back on, then a quiet watch while the agent confirmed the declared state and the running platform finally matched.
6. Re-price the risk. Pruning switched off for long-lived resources, the volume’s reclaim policy moved to Retain, and a pipeline check added that flags any change removing namespaces, persistent volume claims, or resource definitions from the declarations.

From freeze to verified: ninety-four minutes. Data loss: whatever the backup interval had granted. Careers: intact, barely.

What the ledger teaches about audit trails

The most sobering part of the review was not the deletion — it was the realisation that Meridian’s much-admired store history had not predicted it. The commit log showed clean, reviewed changes. It showed intent. What it never showed was the reconciler acting: the prune, the pause, the restoration, the drift it had corrected silently at 3 a.m. every night that week. Kabir’s summary for the compliance committee borrowed the oldest distinction in finance: the store is the order book, but the agent’s events are the trade confirmations, and no audit is complete without both. If you have ever settled an argument with a counterparty by producing the confirmation trail rather than your recollection of instructions, you already believe this. Export the reconciler’s events. Keep them as long as you keep your confirmations. The regulator’s question is never “what did you intend?” — it is “what actually happened?”

The alert that never fired, and the one that does now

Here is a detail from the post-mortem that deserves its own paragraph. During the eleven months before the Friday, Meridian’s monitoring was green. Not mostly green — immaculately green, because the dashboards watched uptime, latency, and order throughput, and none of those break when a configuration quietly diverges from its declaration. The system was unhealthy for nobody’s instruments except the reconciler’s. That has since changed. The desk now alerts on drift itself: any resource that spends longer than a few minutes differing from its declared state raises a flag, regardless of whether the difference hurts performance. Think of it as the difference between checking a stock’s price and checking its delivery versus demat status — one tells you how things look, the other tells you whether the paperwork matches the holdings. Most outages in a reconciled platform are paperwork problems first and technology problems later, and paperwork problems deserve their own alarm, not a footnote under the latency graph.

Position-sizing the remaining GitOps risk

Nobody at Meridian concluded the model was wrong, any more than a stop-loss being hit means position sizing was a mistake. The concentration of power in the store is the feature and the risk at once: the same reviewed file that prevents unlogged server edits can, through one careless conflict resolution, command the deletion of a namespace. Size the exposure accordingly. Reviews are the circuit breakers — keep the diffs small, one intention per change, so a reviewer can actually see the settlement file disappearing. Prune guards are the margin requirement: long-lived resources should be expensive to delete, not cheap. And rehearsals are the simulation trading: Meridian now runs a quarterly drill where they break a namespace on staging and walk the six steps, because the first time you follow an emergency runbook should never be during an emergency. Traders who want the adjacent discipline will find it in a step-by-step guide to trading discipline, in the psychology behind successful trading, and in the evergreen maths of risk-reward ratios for stock traders — same instincts, different clock speed.

The week after, in Kabir’s words

“We nearly lost the ledger to a tool that was only keeping its promises,” he told the team on Monday, and then made the promise he is probably still keeping: every deletion that automation can perform must be slower, louder, and more reversible than any human error it prevents. That is the whole risk philosophy of GitOps in one line, and it fits on a desk sticky note next to the older ones. If the mechanics interest you — the pull model, the reconciliation loop, the guardrails Meridian now swears by — the clearest long-form explanation we know is this GitOps walkthrough, which covers the failure modes with more patience than a Friday afternoon deserves. Reconcile daily. Guard the deletions. Keep the confirmations. Your future auditor — and your future self — will bill you less for both.