Service billing nightmare: a cautionary tale
It was supposed to be $20/month. But this month it was over $4000. What happened?!
---
Just recently I helped a client deal with this service billing horror. I think it highlights an important lesson for all of us.
It goes like this:
Client has a subscription to an outside service that's priced at around $20/month.
All's well for a few couple of years.
But one day Client gets an invoice from this service for over $4000 in charges — for a single month!
It doesn't even seem possible — but it is.
The service was a reputable outbound email provider (like SendGrid, SparkPost, or whomever you're using to handle outbound email sent from your live site).
There was no shady practice going on — not by Client, and not by the service provider.
But somebody was being very shady.
Somebody got their hands on Client's private API key for this service.
Somebody used that API key to send millions of spammy emails through Client's account, within just a couple of days.
And the billing nightmare?
Well, Client's service plan was priced so that they were indeed charged just $20/month to send up to X thousand emails — and then for any additional email messages over that limit, they'd be billed just a fraction of a cent each.
They'd never actually exceeded that "X thousand" emails limit. So they happily paid the $20 each month.
But once the baddies went to town on their dime, the "faction of a cent each" charges added up very quickly.
The result?
Client is on the hook for over $4000 in usage fees.
Client must deal with the damage to their domain's email sender reputation — since their account just launched millions of "This one bedroom trick changed my life" emails (and worse).
Client can't send any outbound emails from their live site at all — because the outbound email service has suspended Client's account for malicious activity.
So: how did the bad guys get that API key?
It turns out, Client was running a clunky home-grown backup system on their live site, which resulted in all site files (including one containing the API key) being downloadable by anyone on the internet.
Once the spammers managed to stumble across that file (likely through a bot that simply downloads everything it can find, across as many websites as it can crawl through), they had everything they needed to a) make the Client miserable and b) spam the good people of the world with annoying sales emails.
Here's the lesson:
Well, there are many. I may try to get into those more in a separate email.
But for now, the lesson is this:
Be aware of where and how you store your account credentials. And that's not just passwords for your website. Email passwords and third-party service API keys are part of the picture too.
In the end:
We were able to communicate with this email provider, and — with a lot of convincing — they agreed to waive the overage fees (so $4000 in billables literally just went away) and to reinstate Client's ability to send emails again.
Any damage to their email sender reputation was already done and beyond anyone's control, but even that began to fade away quickly once we got things locked down again.
So they were lucky. In the end, they got off light.
But of course, luck is not a plan.
Be careful out there, folks.
All the best,
A.