I’ve been working in IT for a long time now, which means I’ve accumulated a respectable collection of mistakes. Some were funny. Some took production down. Some cost money. And some taught me lessons that I still carry into every system I build today.

The interesting part is that the mistakes changed as I did.

When I was starting out, I could destroy something because I didn’t understand a terminal command.

Years later, I was making mistakes in queues, distributed systems, storage architecture and production economics.

Apparently becoming a more experienced engineer doesn’t mean you stop making mistakes.

You just unlock more sophisticated ways to break things.

I blew up a power supply on my first day

My first IT-related job started while I was still in college.

I worked as a lab assistant, surrounded by old computers and hardware that had probably survived several generations of students.

On my very first day, I found an old power supply with one of those physical voltage selectors on the back.

115 V. 230 V.

For reasons that seemed reasonable to my teenage brain, I switched it from 230 V to 115 V while it was still connected to the mains.

There was a bang. And smoke.

My supervisor interrupted whatever he had been explaining to the class, looked at the dead power supply and said something along the lines of:

“And that is how you destroy a power supply older than this rhinoceros. By the way, smoke from exploded capacitors isn’t something you want to breathe.”

“Rhinoceros” was one of the affectionate names we received when making particularly intelligent decisions.

A wonderful beginning to my career.

Lesson #1: electricity is remarkably unforgiving of curiosity without attention.

Then a friend taught me rm -rf

Not long after that, I was learning how to set up a LAMP stack: Linux, Apache, MySQL and PHP.

I knew almost nothing.

So I did what every promising junior engineer does: I found someone who knew more than me and bombarded him with questions until he apparently decided violence was the only remaining option.

At some point he suggested:

rm -rf /*

I ran it. Of course I ran it. I didn’t understand what it did. Three days of my work disappeared.

In retrospect, this might have been one of the cheapest and most valuable lessons I’ve ever received.

Lesson #2: Never, ever paste a terminal command you don’t understand. Especially when someone who has clearly become tired of answering your questions gives it to you.

“ROOOOMA, THE INTERNET IS DOWN!”

A little later I was working as a system administrator.

One day I needed to SSH into the central office server and change some network configuration.

Something wasn’t working correctly, and I concluded that I needed to restart the network interface.

Simple enough.

I searched for the commands: disable interface, then enable interface.

So I disabled it.

And then I stared at my terminal.

My SSH session had disappeared.

There was a very short and beautiful moment in which my brain reconstructed what had just happened.

I had been connected to the server through the network. And I had just disabled the server’s network interface.

The server was physically locked inside a rack.

I didn’t have the key.

And that server happened to be rather important to the office network.

Then I started hearing it from across the room:

“ROOOOOMAAAA! THE INTERNET IS DOWN!”

That was approximately when the theoretical consequences of my command became extremely practical.

Eventually the problem was fixed.

And that day I ceremonially received my own key to the server rack.

Apparently management concluded that if I was going to break the network remotely, I should at least be able to walk over and repair it locally.

Lesson #3: think about the consequences before pressing Enter, not immediately afterwards.

The server with root / password1234

The same company gave me another lesson, this time about trust.

One day the director came to me and said:

“Some guys are coming today to install a new server for accounting. Open the server room and give them whatever they need.”

Fine.

They arrived, installed their server, and I gave them an IP address and whatever network access they required. Then they left.

The next morning I was the first person in the office.

I turned on my computer and immediately noticed something strange.

The network technically worked. The internet technically worked. But nothing actually worked: ping and traceroute were fine, while opening a website or connecting to an FTP server was nearly impossible.

I went into the server room and checked the gateway.

There was an absurd amount of traffic coming from yesterday’s new server.

I unplugged its network cable. Everything instantly returned to normal.

Interesting.

I checked the logs.

They were basically full of:

ping 8.8.8.8

It looked suspiciously like someone was using our shiny new accounting server as part of an attack.

So I contacted the people who had installed it.

Their response was essentially:

“We don’t know. We’re out of town. Here’s the login and password, take a look.”

The credentials were root and password1234.

Right. Mystery solved.

The server had barely entered the public network before some completely ordinary automated bot found it and compromised it.

That incident changed how I thought about responsibility.

It didn’t matter that I hadn’t configured that server.

It was inside infrastructure I was responsible for.

Give systems and people only the permissions and resources they actually need. And don’t outsource your responsibility along with the work.

Years later, my mistakes became more expensive

Fast-forward several years.

I was building Romergo and needed infrastructure for user-uploaded images. At the time I chose Cloudinary, and it was a great fit.

One particularly useful feature was the ability to transform images through URLs. I built the backend integration, rewrote the frontend around it, and everything worked beautifully.

Until users arrived.

Which, as it turns out, is when software becomes considerably less predictable.

One user encountered a frontend bug where a newly uploaded image didn’t immediately appear.

Their perfectly reasonable response was to upload it again. And again. And again. More than a hundred times.

My perfectly unreasonable architecture happily accepted every copy.

So I added deduplication.

Images received hashes, and if the same image already existed inside the project, I reused the existing asset instead of storing another copy.

Problem solved.

Sort of.

Then Romergo started costing more than its users

My next assumption was even more interesting.

I had designed the project during a period when approximately one and a half people were using it.

At that scale, almost every infrastructure decision is cheap.

Then usage started growing.

And I discovered something I hadn’t modeled properly: bandwidth wasn’t free.

Cloudinary’s pricing model made sense for its product, but not for the economics I wanted for mine.

My philosophy for personal products has generally been:

free for as long as reasonably possible, and extremely cheap when free is no longer possible.

Suddenly image infrastructure was violating that rule.

The service was already in production, people were actively using it, and serving them was becoming more expensive than I was comfortable with.

There were two problems: one needed solving right now, and the other needed solving properly.

For the immediate problem, I restricted upload sizes and became much more aggressive about caching.

That bought me time.

Then I started redesigning the storage layer.

Eventually I moved the workload to Cloudflare R2 and changed how assets were delivered.

The economics became dramatically better for my particular workload.

But the more important lesson wasn’t “Cloudinary bad, R2 good.”

Cloudinary had done exactly what it was supposed to do.

I had designed the architecture without sufficiently designing its economics.

That was the bug.

Architecture isn’t only APIs, databases and beautiful diagrams.

Cost is part of architecture too.

Then I accidentally messaged more than 500 people

Signailist gave me several more opportunities for personal growth.

It’s a system that processes information, transforms it and automatically publishes content.

Which means there are certain checkboxes you should be particularly careful with.

One of them was essentially:

Send to all contacts.

While testing automatic publishing one day, I forgot to turn it off.

I published my test.

And then realized what I had done.

The message went to more than 500 of my contacts.

There is a special kind of silence that occurs after pressing a button and suddenly understanding that the button cannot be unpressed.

That was one of those moments.

Nothing teaches you to respect bulk operations quite as effectively as accidentally performing one.

After experiences like that, confirmations, previews, dry runs and recipient counts stop feeling like unnecessary UX friction.

They become safety equipment.

One customer was stronger than a server built for dozens

My second memorable Signailist failure was less embarrassing and much more interesting technically.

I had designed message processing around queues.

On paper, everything looked reasonable.

Tasks entered the system, workers processed them, and results went elsewhere. Beautiful.

Except I hadn’t thought deeply enough about what happened when the amount and shape of data changed.

Some messages could enter the queue multiple times. Work wasn’t grouped into batches efficiently enough. Thousands of tiny operations created overhead that wasn’t obvious when looking at any individual task.

Eventually one customer managed to put enough pressure on a DigitalOcean server intended to handle dozens of customers that the system started struggling.

One customer. A server for dozens. Impressive, really.

The individual operations weren’t particularly expensive.

There were simply far too many of them.

That changed how I think about backend systems.

It’s easy to reason about:

task → API → result

But production systems also have another dimension:

data volume → duplication → batching → concurrency → queues → resource pressure

A thousand tiny jobs can sometimes hurt much more than one obviously expensive job.

Signailist is still one of my favorite projects precisely because of this.

It’s complicated and autonomous. Information flows through it without someone manually supervising every step.

And systems like that are extremely good at discovering assumptions you didn’t realize you had made.

I still make mistakes

I could probably write another article with the ones I’ve forgotten.

And I’m sure I’ll produce several exciting new ones in the future.

But looking at these stories together, there’s something I like about them.

My first mistakes taught me individual rules:

  • Don’t touch electrical systems carelessly.
  • Don’t execute commands you don’t understand.
  • Don’t disable the network you’re currently using to access the machine.

Later mistakes taught me something broader:

Design for failure.

  • Assume users will click twice.
  • Assume jobs will be duplicated.
  • Assume traffic will grow.
  • Assume a third-party service will eventually become expensive.
  • Assume credentials will leak.
  • Assume the person operating the system will make a mistake.

And assume that person will occasionally be you.

Experience didn’t make me stop making mistakes.

It mostly changed what happens after I make one.

I recover faster. I understand the blast radius better. And whenever possible, I change the system so that the exact same mistake becomes harder to make twice.

Maybe that’s one of the less glamorous definitions of becoming a senior engineer: you don’t run out of ways to fail. You just build increasingly good guardrails from the wreckage.