Downtime at 3am
This is a little silly, but I'll share it, because it might matter for you.
I have several online systems that help me run my business, and early on I found they'd sometimes go offline -- and stay that way until I noticed and rebooted them. I was not happy.
Lesson learned:
Systems go down.
When they do, I need a way to respond quickly.
So I arranged for monitoring that would send me a text message if anything important went offline.
Once in a few months or so, I'd see a downtime message, take corrective action, and be happy.
But then I noticed that if a text came in while I slept -- like at 3am -- I simply wouldn’t hear it. So a system might stay offline for hours. I was not happy.
Lesson learned:
Timely alerts need to be noisy,
interruptive,
hard to ignore.
So I arranged for downtime alerts to come in through a dedicated phone app with attention-getting alert sounds.
Once in a few months or so, I'd be awakened by an urgent alert, take the same old corrective action, and be happy.
Recently, I’ve found a couple of those systems triggering late-night alerts multiple times a month. I was not happy.
Lesson learned:
The downtime was unpredictable,
but the fix was always the same.
So I arranged for an extra step in my downtime monitor: before alerting me, try the standard fix a couple of times (and email me about that), and only alert me if that doesn't solve the outage.
This morning, I found an email from the monitor.
It had solved the problem while I was asleep.
All systems were live, and the downtime was less than a minute.
Maybe sometime I'll explain why this is a little silly (but only a little).
But for now, I think there's a winning pattern here:
Making everything run perfectly all the time
is an expensive proposition.
Often it's enough
to ensure you notice when something's amiss,
and to take corrective action.
And when the patterns reveal a consistent fix
for a common problem that's keeping you up at night (literally or figuratively),
automating that fix is surely worth the effort.
All the best,
A.