MOTOSHARE 🚗🏍️

Turn Idle Vehicles into Income

Owners Earn. Riders Move. Motoshare Connects.

Start with Motoshare

Four Alert-Tool Failures and the ServerlessOps Fix for Each

Four Alert-Tool Failures and the ServerlessOps Fix for Each
Trading Technology

Meera, a part-time market watcher in this story, is a composite character invented for illustration. She keeps a small set of cloud functions that read price feeds, check her own rules, write entries into a journal spreadsheet and send her a phone alert when a rule triggers. One Tuesday she received the same alert four times in a minute and, a week later, none at all during a busy session. Her instinct was to blame the data provider. The real cause sat in her own plumbing, and fixing it is a tidy introduction to ServerlessOps. This article is about software reliability for personal tooling; it offers no view on markets, securities or what anyone should do with money.

Symptom one: the alert that arrived four times

Meera’s alert function did three things in order: it composed a message, sent it, and then marked the rule as handled. Occasionally the sending step succeeded but the final marking step ran past its time limit. The platform, doing what it is designed to do, treated the run as failed and started it again, and again.

Diagnosis: the function was not safe to repeat. Event platforms generally promise to deliver an event at least once, which is a sensible choice for the provider and a trap for anyone who forgets it.

Treatment: before sending, check a small table for an entry that identifies this particular alert. The key should describe the real-world fact, say the rule name plus the date and the price level crossed, and should not be a message number issued by the platform, since a replayed message gets a fresh number. The Ops Atlas makes exactly this distinction in its ServerlessOps write-up, and backs it with a simulation in which logging attempts alone still left hundreds of double effects, while logging plus a business key left none.

Symptom two: the silence during a busy session

The silent session was stranger. Nothing in her error graph moved. When she finally opened the holding queue she had set up months before, it held a long line of unprocessed items. They had been sitting there since the busy session began.

Diagnosis: a burst of price updates had started many copies of the function at once. They all hit the spreadsheet service, which has modest limits, and most calls were refused. The functions failed, retried, failed again, and finally dropped into the holding queue. Meanwhile the error alarm she had configured counted errors on the function but not the age of what was stuck.

Treatment: two changes. First, a ceiling on how many copies may run simultaneously, set below what the spreadsheet service tolerates. A caps-first mindset is what the Atlas calls out when it lists uncapped concurrency among the anti-patterns, since a spike that sails past a fragile downstream can become an outage there. Second, an alarm on how long the oldest queued item has been waiting. Age tells you something a count cannot.

Symptom three: the bill that crept up

Meera had assumed a hobby project would cost pennies. The monthly charge rose unexpectedly during a week in which her spreadsheet service was slow. Billing on these platforms follows running time, so a call that normally finishes in a fraction of a second but now takes two seconds keeps the function, and its bill, alive far longer. In the Atlas’s lab, a tenfold slowdown of a dependency raised one function’s cost by roughly nine times and pushed parallel runs close to the account’s ceiling.

Diagnosis: a slow downstream turned into a cost problem and a capacity problem at once.

Treatment: track cost per million events next to the success rate, and give the downstream call a shorter timeout than the function itself, so a sulking service is abandoned early instead of waited on.

Symptom four: retries stacked on retries

When Meera reviewed her setup she found retry logic in three places: the feed reader, the function and the alert sender. Each seemed modest. Together, they meant one failed fetch could spawn a pile of attempts. The Atlas quotes advice from an Amazon engineer that retrying at every layer multiplies load alarmingly, so the guidance is to retry in one chosen place and let the others fail fast.

The discipline habits behind the fixes

  1. Treat repeats as normal. Design every side effect to survive being run twice.
  2. Give failures an owner. A holding queue nobody reads is just a delayed loss.
  3. Cap parallelism. Know what your slowest dependency can take.
  4. Measure outcomes. Watch real success, duplicate effects, item age and unit cost.
  5. Rehearse. Replay a block of test alerts and confirm nothing doubles.

Calm, written-down routines also underpin good trading habits, which is why our step-by-step piece on building discipline into how you handle your trades sits comfortably beside this one, and the guide to a settled start to the trading day makes a similar case for routine over reaction. Treat both as background reading, not as prompts to act.

What she wrote down afterwards

Meera kept a single page for each function: its trigger, what it changes, who is told when it stalls, its time limit and the limit on its slowest call. Six months on, that page saved her an evening when a feed changed its format, because she could see at a glance which functions touched the journal and which merely read prices.

A ledger comparison

The household-ledger version of all this is simple. A cheque that clears twice is a duplicate side effect. A pile of unopened bills is a holding queue without an owner. A card with no limit is uncapped concurrency. If you would not run your finances that way on paper, there is little reason to run your automation that way in the cloud.

Meera’s tools now send each alert once, tell her when something is stuck, and cost about what she expected. Her setup is still small; the difference is that it is predictable. That, more than any clever feature, is the goal of the discipline. Remember that this is an illustration about software, not financial guidance, and decisions about money belong with you and, where appropriate, a licensed professional.

0 0 votes
Article Rating
Subscribe
Notify of
guest

0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x