Meridian Broking is a fictional mid-sized brokerage, invented for this story but staffed with people you will recognise: Vinay, head of operations, twenty years of reading charts and bad excuses; Divya, compliance analyst, allergic to unwritten promises; Priya, junior analyst, fastest ticket-closer on the floor. Last year a vendor walked into their boardroom to sell them AIOps for Security — machine-learned alert scoring, a language-model assistant, and eventually, auto-closing. What unfolded over the next quarter is the most honest story we know about how a careful team should meet this technology, and it reads the way market people think: in probabilities, samples, and position sizing.

Demo day: what AIOps for Security promised Meridian

The demonstration was genuinely good. On screen, the agent worked through forty pending alerts, summarised each one in plain language, and closed the routine ones — no waiting, no coffee break. The account executive, Sanjay, put a number on the slide: seventy percent of manual workload removed, accuracy figures to follow in the appendix. Priya was impressed. Divya was already reading the appendix, which is where she found numbers quoted “from deployments” with no base rates attached and no per-category split. Vinay asked the question he asks every vendor, borrowed from his trading years: “We live and die on false signals. What does your system miss, how often, and how would we even know?” Sanjay’s answer — “the model is highly accurate” — was the moment Divya started taking notes in earnest, because it was not an answer to the question that was asked. Pressed politely afterwards, Sanjay offered a case study from “a similar-sized financial firm”, which turned out to be a different country, a different mail stack, and an appendix with averages where categories should have been. Priya asked, genuinely puzzled, why accuracy on a demo meant anything at all for their queue. Nobody had a good answer, which was itself educational.

Divya’s counter-proposal: a blind-label quarter

Her proposal to the committee was modest in scope and firm in structure. One category only — suspicious-email reports, the highest-volume queue they owned. The agent would run in observe mode for the quarter: ranking, summarising, suggesting, but touching nothing. Meanwhile Divya and Priya would blind-label a sample of the same cases — verdicts written down before anyone saw what the agent said, with a colleague holding the key until labelling finished. The mechanics were unglamorous: a shared spreadsheet, a case number, a true-or-false column, and a rule that disagreement between the two labelers went to a third colleague for a deciding read, never a coin toss. Vinay approved it in eleven minutes and asked one thing: “Show me the miss rate the way I’d want it shown for a strategy — with a confidence bound, not a vibe.” The committee had heard AIOps for Security pitched three times that year; this was the first time anyone had asked it for a position-sized argument.

What the quarter produced

The labels came back in April: 1,240 emails adjudicated, 71 of them genuine threats, and the agent — running quietly alongside — had missed 2 of those 71. Divya laid the numbers out for the committee without decoration. The headline miss rate was 2.8%, which sounds acceptable until you price the denominator: with only 71 true cases, the 95% upper bound on the miss rate sat near 8.6%, roughly four times the firm’s 2% requirement for any auto-closing privilege. By the gate Meridian had written, the agent failed — not because it was bad, but because 71 examples cannot certify a number that small. There was good news in the same table and Divya gave it its due: the agent correctly identified 69 of 71 genuine threats, closed with 99.8% precision, and had compressed hours of daily reading into minutes. The failure was specific and therefore fixable — more labels, more genuine cases, a tighter bound next quarter. Vinay’s summary entered company folklore: “That’s a strategy with two losing trades out of seventy-one. You don’t size up off that; you gather more data.”

Reading the bound like a trader

The team’s instinct here is worth borrowing, because it is exactly the discipline that separates systematic traders from gamblers. Anyone who has studied how MACD readings can mislead a beginner knows that a signal can look immaculate on the chart and still fail half the time at the margins — the sample around the signal decides its worth. The same arithmetic governs automation claims. Meridian’s quarter had produced flattering numbers too — its agent’s verdicts on which alerts were genuine ran 86% precise — and both of those were beside the point: the gate question is what the machine waves through, and that number needs far more genuine cases before it can certify anything. A team that has absorbed the classic intraday mistakes worth avoiding will recognise the failure mode instantly — acting decisively on evidence too thin to support the position. Zero misses would not have saved the claim either; a clean sheet on a small sample bounds the true rate near 3 divided by the sample size, which is a ceiling, not a compliment.

What Meridian granted the AIOps for Security agent, and what it withheld

The committee’s decision, written down in three lines, was more interesting than a yes or a no:

• Grant: recommend level only — the agent summarises, ranks, and drafts dispositions, while humans click the close button.
• Withhold: auto-close, pending a labelled true-positive sample large enough to certify a 2% miss bound with the canary programme running.
• Maintain: fifty planted canary cases per quarter; if the machine closes three of them, the closing privilege dies that day — no meeting, no appeal.

The workload arithmetic had earned the grant honestly: 93.5% of the queue’s reading burden was judged removable on the labels, which meant Priya’s evenings came back without anyone pretending the machine was infallible. What was withheld was closing privileges, pending a bigger sample — Divya estimated the true-positive count would need to reach into the hundreds before a 2% bound could clear, with 236 reviewed cases as the rough entry ticket once a miss appeared. One clause surprised Sanjay: those planted cases. He asked whether the three-strikes withdrawal was standard industry practice. Divya said it was now.

The habit that outlived the pilot

Meridian’s story ends where good tool stories should: with the tool working, the humans doing different work, and the paperwork describing reality. Vinay kept the bound arithmetic on one laminated card next to his risk tables — rule of three, sample floors, the 2% gate — and visitors sometimes photographed it. Priya ran the canary programme and found, to her surprise, that designing a realistic fake phishing case taught her more about attacker tradecraft than a year of closing real ones ever had. Divya’s one-line rule went up on the wall of the security closet: “Numbers without denominators are marketing.” The retest cadence got a calendar entry — labels refreshed each quarter, the bound recomputed, the gate re-signed — because everyone had absorbed Vinay’s other market lesson: correlations decay, and so do accuracy claims, the moment the adversary changes strategy or the mail mix shifts. A second category, account-takeover alerts, entered the observe pipeline in June with the same structure and none of the drama, which is what maturity looks like in practice. The team’s next vendor meeting was shorter than the first — the demos had not changed, but the questions had, and that turned out to be the whole difference. Sanjay, to his credit, came back with per-category numbers unasked. Anyone planning their own evaluation would do well to keep the atlas write-up on AIOps for Security beside Divya’s rule; between the two, most vendor claims can be sorted in an afternoon. And when the question widens from security tooling to software in general, the team’s approach matches researching software with the care of a money decision — same scepticism, same sample-size instinct, fewer regrets.

A fictional firm, yes. The arithmetic is real, the gates are real, and the outcome — cautious adoption with receipts — is what the evidence supports. Markets taught these habits first; security operations are only the latest students.