A real security audit of OpenLedger had been sitting on my to-do list for a while. Right now our only users are people I actually know personally, early testers, pilot partners, folks I brought on myself. Nothing's really at stake yet if something's wrong. But once real strangers start trusting us with their contracts and their data, I don't get to find problems this casually anymore. So I wanted to get ahead of it while the cost of being wrong was still zero.
Then something happened that looked, for a few hours, like the thing I was trying to get ahead of.
The morning it all kicked off
I woke up to an email from SendGrid warning that our credits were running low. Odd, since we don't send that much volume. I poked around and noticed some genuinely weird signup activity, and my first thought was: are we being spammed?
Rather than dig in first, I figured, whatever, it's probably time to upgrade to the Pro plan anyway. I didn't want a credits issue to accidentally block real users from getting their emails. So I upgraded, then started deploying fixes: domain validation to block disposable and placeholder addresses, rate limiting on the signup endpoint, rotated API keys as a precaution.
More emails kept coming in. Then I fixed a separate Redis issue that had been quietly misbehaving. And that's when things actually started running wild, thousands of signup notification emails, all addressed to things like test@example.com, bob@example.com, alice@example.com, all firing within about ten minutes of each other.
So naturally: I went to delete these fake spam users from the database. Except I couldn't find them. Not one of them existed in production. But I'd definitely gotten thousands of emails about them.
Emails were sent. No users were created. That's a genuinely confusing thing to stare at.
The actual answer, and the part that should have been obvious
Here's the thing about debugging your own code: sometimes the answer is embarrassing.
A few days before all this happened, I added a feature I was quietly proud of — a hook that fires every time a new user is created and sends me a notification. "New signup on OpenLedger." Simple, useful, took about ten minutes to build.
What I forgot is that my test suite creates users too. Fake ones, with names like bob@example.com, alice@example.com, owner@example.com, viewer@example.com. They exist for about a millisecond during a test run and then get deleted. I've been running these tests for months without thinking about it.
The moment I added that notification hook, every single test run started quietly queuing email jobs in the background — one for each fake user created. The jobs piled up in Redis over several days of active development. When I fixed the Redis issue that morning and the worker came back online, it flushed the entire backlog at once. Hundreds of notification emails, all timestamped with the moment they finally ran rather than when the tests originally created the users, all for users that no longer existed in the database.
I built a "tell me about every new user" feature and forgot I was a user myself. Or rather, that my test suite was.
The funniest part of the whole thing
While all this was unfolding, I happened to be watching our SendGrid sender reputation score in real time. It had been sitting somewhere in the 70s. As the flood of emails went out, I watched it climb, live, up to 91.
Turns out the reason is almost absurd: sender reputation is largely based on how many of your emails actually land versus bounce. Since the flood was hitting real, legitimate domains (even if the addresses themselves were fake test accounts), the delivery rate looked great to SendGrid, and our score went up because of it. Watching it tick upward in real time, in the middle of what I thought might be a crisis, was one of the funnier things I've experienced building this.
While we were looking, we found the real stuff
Chasing down that mystery meant going through our server logs properly for the first time, and that turned up three genuinely separate things that had been running in the background the whole time:
- A slow-drip registration bot, cycling through scraped email addresses and submitting them to our signup form over about a week and a half. Low volume, easy to miss if you're not looking.
- A credential scanner, probing for exposed environment files and API keys across hundreds of common paths. Found nothing, because we don't serve files that way.
- The standard background noise every public web app gets: bots hunting for old PHP vulnerabilities and WordPress installs we don't have. Irrelevant to us, but constant.
None of it succeeded. No user data was touched, no unauthorized access happened, nothing was ever exposed. But we tightened things up anyway: proper rate limiting, better domain filtering, a deploy script that was quietly sending more emails than it should have, and a few smaller hardening steps.
Why I'm posting this
OpenLedger's whole premise is that people deserve real visibility into what's actually happening, not vague assurances. It would be a little hypocritical to hold my own build to a lower standard than I'm asking clients and contractors to hold each other to.
So: our first security audit is done. Nothing was actually broken by an outside party. The scariest-looking incident of the bunch turned out to be a feature I built myself, three days earlier, that I'd already forgotten about. Everything's patched, tightened, and we're back to normal, with a slightly baffling reputation boost to show for it.
Onward.