Few moments feel as exposed as the minutes when a trading screen stops updating while the market keeps moving. Orders hang, quotes freeze, and everybody in the room starts offering a different theory. This guide shows, step by step, how a small trading or brokerage-technology team can respond to that kind of disruption using the habits grouped under the name IncidentOps. It is about operating a platform, not about trading decisions: nothing here is investment advice, and no strategy or security is recommended.

Our outline leans on the Ops Atlas reference page for IncidentOps, a source that frames the topic as joined-up handling of disruptions nobody planned and admits the name is casual. We adapt its ideas to the pace of market hours, where minutes matter more than they do on most websites.

Step 1: Decide in advance what counts as an incident

Before any market opens, write down your triggers. A frozen price feed, rejected orders for more than one client, or a login failure that persists past a set number of minutes are typical examples for a trading platform. The Atlas points to triggers used at Google as a model: the problem requires another team, customers can see it, or it is still unsolved after about an hour. On a trading desk you would shorten that last clock considerably, but the principle holds. Written triggers mean nobody has to wonder whether it is “bad enough” while the clock runs.

Add a severity scale of three or four levels, with one sentence for each. Your top level might be “clients cannot place or cancel orders.” Keep it short enough to remember under stress.

Step 2: Name the roles and keep them separate

The next step is to decide who does what. The commander coordinates and decides; this person is deliberately not the one debugging. Only the operations lead and the engineers that person picks may touch the live system. A communications lead tells clients and colleagues what is happening, and a scribe records the timeline. If the team is small, one person can wear two hats, but never the commander and the fixer together.

The Atlas stresses that these are hats, not titles, and recommends rotating the commander role. The hat falls to the most senior person present on that call, whatever they do day to day. Practise with colleagues in quiet periods so the first time is not during a volatile session.

Step 3: Declare, then size up

When a trigger fires, say it plainly in the response channel: “Declaring an incident, severity one, order entry failing.” The commander then sizes up the situation by asking what is affected, who knows, and what changed recently. One published playbook describes the loop as size up, stabilise, update and verify, repeated until things settle. Having a loop to follow gives a nervous team something to hold on to.

Step 4: Stabilise before you explain

The instinct in a freeze is to work out why it happened. Resist it. The Atlas lists generic mitigations that need only a location for the fault: roll back the last release, drain a faulty node out of rotation, switch off a recently added feature, fail over to a standby, or shed part of the load. If a deployment went out shortly before the freeze, rolling it back is usually the first move.

This is not laziness. The Atlas recounts a cluster failure that dragged on for over six hours; going back to earlier images the moment the fault location was known would have ended it roughly two hours sooner than the fix that finally landed. Understanding comes later, with the evidence preserved.

Step 5: Communicate on a clock

Clients tolerate bad news better than silence. Give the communications lead a fixed rhythm: the opening message about five minutes into the call, a scoping note soon after, and after that a fresh note every twenty minutes or sooner while the first two hours last. Use plain status words, moving from investigating to identified to monitoring and finally resolved. The closing message should say whether any data or any pending orders were affected, in factual terms and without speculating about market outcomes.

Make sure the status page and the paging path do not depend on the systems that failed. The Atlas gives a 2017 cloud outage as a reminder: the provider’s own service-health console went unusable for hours because it depended on the broken service. If your client notices live on the same infrastructure as your trading engine, they will vanish when you need them.

Step 6: Hand over explicitly

Market-hours incidents can be long, and people tire. When a commander is relieved, the exchange should be spoken: the outgoing person says “you are now the commander, okay?” and waits for a clear reply. The Atlas recommends exactly this wording style. It prevents the dangerous middle ground in which two people each assume the other has the reins.

Step 7: Review within days and keep the list short

Settle on who will own the review while the call is still running. The playbook the Atlas points to sets a deadline: a written review inside three calendar days after the most severe incidents, and inside five working days after the tier below. Ideally the facilitator sat out the hands-on work, which lets the questions come from outside the heat of the moment. End with a few actions, each having one owner and one due date, and look for system conditions rather than a person to blame. Quoting classic safety research, the Atlas reminds us that hindsight biases the judgement of human performance after an accident.

Step 8: Measure carefully

It is tempting to boil everything down to one average recovery time. Be wary. Incident durations are skewed, and a single long event can swing an average dramatically. The Atlas works through an eleven-incident sample where the middle value for mitigation time was 32 minutes but the arithmetic mean reached 78.4, with one top-severity case alone responsible for 45.5 minutes of the shift. Track distributions, minutes of error budget used, how heavily the pager was hit, and how many actions closed on time instead.

Step 9: Plan the first twenty minutes in detail

Most of the damage in a market-hours freeze is done early, so script that window. In minute one the person who noticed posts in the response channel. By minute three a commander has acknowledged. By minute five the communications lead has sent a holding message. Between minutes five and fifteen the operations lead reports on what changed recently and the commander decides between a rollback, a failover or a feature switch-off. At minute twenty the next scheduled update goes out regardless of progress, even if it says only that work continues. Having a script does not remove judgement; it removes the dithering that happens when nobody knows who should move first.

Pair the script with one simple dashboard that everyone looks at, so debates about the facts do not eat the clock. Whatever you pick should be reachable even when the main platform is down.

Step 10: Rehearse

A rehearsal in a quiet hour is worth more than any document. Invent a scenario, such as “the price feed freezes but order entry still works,” assign the hats, and run the clock for thirty minutes. Afterwards ask what was unclear. The Atlas recommends including at least one paging drill and one tabletop in a first ninety days.

Tie it into your wider routine

Disciplined preparation is a theme this site returns to often. If you are building your own habits, you may find our write-up on setting up a steady, composed start to the trading day a natural companion, and the step-by-step piece on keeping personal discipline when screens get busy approaches the same theme from the individual’s side. Both discuss behaviour and process in general terms, not specific trades.

A starter checklist

• Written triggers and a four-level severity scale.
• Names for commander, operations lead, communications lead and scribe.
• A status page and paging path that sit outside the trading infrastructure.
• A message template with a fixed update rhythm.
• A spoken handover phrase.
• A review owner named on every serious event, with a short action list.

That list takes an afternoon to draft and one quiet session to test. The next time a screen freezes, your team will already know who speaks, who fixes and who writes things down, and that clarity is the whole point of IncidentOps.