# Roman / h3lpkey — locally hosted full articles Contains 8 complete local article texts. The 15 project articles are linked in https://h3lpkey.com/blog/posts.txt and are not reproduced here. # I wanted to switch my brain off. So I built Holearth. Canonical URL: https://h3lpkey.com/blog/three-days-to-build-holearth/ ← Holearth articles (https://h3lpkey.com/blog/?type=holearth) Holearth · A three-day experiment I wanted to switch my brain off. So I built Holearth. I got bored of repeating levels in a game I played to unwind. Three days later, I had a growing hole, real cities to eat, and a whole planet on the menu. Holearth start screen showing a New York map and the location and hole appearance controls Pick a city, pick an appetite. Map data © OpenStreetMap contributors, ODbL. When I’m tired in the evening, I like playing stupid little games. Eat everything. Clean up the rubbish. Find the matching pair. The kind of thing where my brain can finally clock out. The other evening, I opened Hole.io on YouTube again and realised I’d already played through it. The levels were repeating. I was bored. Well… why not give myself another three-day challenge and build my own game to rot my brain with? :) What if I could eat my own city? Generating a thousand different objects and maps didn’t sound like the most interesting part. But using open GIS data to let someone eat a city they’ve visited, or the neighbourhood they live in? That I liked. That became Holearth (https://holearth.h3lpkey.com/): a browser game where a little hole swallows things, grows, and eventually develops a fairly unreasonable appetite. The easiest part turned out to be getting map tiles, turning building footprints and roads into a playable scene, and filling it with little objects: trees, cars, street furniture, and things to hoover up. The street layout comes from real geographic data. The little objects are procedurally placed around it. I don’t need to invent every new level when the planet already has so many places to borrow. I used Three.js (https://threejs.org/) as the base, shadcn/ui (https://ui.shadcn.com/) for the interface, and sprinkled shaders over the whole thing. A dust vortex lifting cars and debris from a city street in Holearth The dust vortex at street level, picking up cars and anything else in its way. Map data © OpenStreetMap contributors, ODbL. A hungry hole, a lot of tiles Once the hole reached roughly a kilometre in size, the ordinary map started looking a bit dull. So I decided to switch to satellite imagery. Much better. Also, more things to download. Both maps and satellite images have weight. The services supplying them don’t always respond quickly, and sometimes they fail altogether. A game for switching my brain off was giving me plenty to think about. I ended up loading tiles as the player moves, keeping the nearby world ready without trying to download everything at once. The beginning of a round needed a different approach. I prepared map data for 20 starting locations and put it on my own server, so starting a game wouldn’t depend entirely on a third-party service answering at that exact moment. I gave myself a limit: keep the hosted map data under 1 GB. Enough to get people playing quickly, without quietly downloading the whole planet onto my server. And then the planet becomes dinner As the hole keeps growing, the flat map gives way to a full 3D globe. Behind it is a photographic panorama of the real Milky Way. Apparently, eating a few cars wasn’t going to be enough. I also played with different kinds of holes, so I could pick one to suit my mood. Objects react differently depending on which one I choose. It’s a completely unnecessary thing to spend time on, which probably explains why I enjoyed it. At larger scales, the ground itself starts getting torn away, leaving exposed crust behind. I found that strangely funny, and it works especially well with satellite photos: a familiar landscape, with chunks missing because I ate them. For three days, I think this was a pretty good experiment. Complicated, silly, and a lot of fun to build. Now I have something of my own to mindlessly play in the evenings. I’m happy :) Play Holearth (https://holearth.h3lpkey.com/) and see how much of your city you can eat. Thank you to all these sources! This little game borrows a lot from people who make geographic data, imagery, and tools publicly available. These are the public APIs, tile services, and downloadable datasets used in the project: - OpenStreetMap contributors (https://www.openstreetmap.org/copyright) — building footprints, roads, parks, water, and the location-picker map. Map data is available under ODbL; the project uses OSM raster and Shortbread vector tiles. - Overpass API (https://wiki.openstreetmap.org/wiki/Overpass_API) — fallback queries for OpenStreetMap geometry, through the public instances at overpass-api.de (https://overpass-api.de/), Private.coffee (https://overpass.private.coffee/), and maps.mail.ru (https://maps.mail.ru/osm/tools/overpass/api/interpreter). - Protomaps (https://protomaps.com/) and Source Cooperative (https://source.coop/protomaps/openstreetmap) — the OpenStreetMap PMTiles archive used to prepare the hosted starting areas and low-zoom world map. These are prepared snapshots, rather than live maps. - Natural Earth (https://www.naturalearthdata.com/) — public-domain world geography for the planetary view. - EOxCloudless by EOX IT Services GmbH (https://cloudless.eox.at/) — global satellite imagery via its tile service. Contains modified Copernicus Sentinel data 2016. - Copernicus Sentinel-2 / Element 84 / AWS Open Data (https://registry.opendata.aws/sentinel-2-l2a-cogs/) — the imagery used for the prepared Tbilisi region. Contains modified Copernicus Sentinel data 2025, cropped, reprojected, and converted to WebP. Satellite images are snapshots, not live views. - Mapzen / Tilezen Terrain Tiles on AWS (https://registry.opendata.aws/terrain-tiles/) — elevation tiles for terrain shading and relief. Thanks to USGS and the other elevation providers listed in the terrain dataset credits (https://github.com/tilezen/joerd/blob/master/docs/attribution.md). - NASA / JPL SRTM, via Sean Bradley (https://sbcode.net/topoearth/srtm-heightmap/#license), and NASA Blue Marble, via Sean Bradley (https://sbcode.net/topoearth/blue-marble-texture-5400x2700/#license) — the prepared global relief and colour textures. - Open-Meteo (https://open-meteo.com/) — the weather API used for local weather conditions. - Municipal Services Development Agency / NSDI Georgia (https://nsdi.gov.ge/en/geoportal) — Tbilisi district boundaries. - ESO/S. Brunier (https://www.eso.org/public/images/eso0932a/) — the Milky Way panorama, under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Resized, converted to WebP, and dimmed in the game. Thank you to everyone who builds and maintains these sources. My evening brain rot has a surprisingly impressive foundation. Keep exploring Play Holearth (https://holearth.h3lpkey.com/) My other three-day game experiment (https://h3lpkey.com/blog/three-days-to-build-the-last-hearth/) All posts (https://h3lpkey.com/blog/) --- # Why I built a tarot bot Canonical URL: https://h3lpkey.com/blog/why-i-built-a-tarot-bot/ ← Father Tarot articles (https://h3lpkey.com/blog/?type=father-tarot) Father Tarot · Experiments that stick around Why I built a tarot bot I wanted an excuse to try the Notion API and build a Telegram bot. Two years later, that little experiment gave me a reason to connect two of my projects. Illustrated tarot cards beside an open notebook and a phone displaying a tarot bot, with gold threads connecting them to a publishing workflow on a monitor From a notebook of card meanings to a bot, and then a connection between two personal projects. About two years ago, I discovered the Notion API. By then, I’d already spent several years keeping pretty much my whole life in Notion: tasks, plans, and a personal journal. Finding out I could build something on top of it immediately made me want to try. A few people I knew were into tarot. And tarot seemed like an interesting fit for the experiment: established card meanings, familiar spreads, and plenty of descriptions available online. What if I put that material into Notion pages and gave an AI access to the database? Notion as the control panel The idea was to generate readings from content that a person could easily edit. Someone without any development experience could adjust the descriptions, change the wording, or add their own interpretation without touching the code. The database could hold descriptions in the language they wanted. They could swap out the card images. The same basic setup could become someone else’s tarot bot. I even thought it might be fairly easy to sell as a white-label solution: give someone the bot and a familiar place to manage its content, then let them make it their own. That was part of what appealed to me about Notion. I already liked using it for myself, and here it looked like a practical editing interface for someone who would never want to open a code editor. I also wanted to try building a proper Telegram bot. I’d made Mini Apps before, but those had mostly been simple web apps wrapped for Telegram. This time I could experiment with buttons, a conversation flow, and even monetization inside the bot. So I built the first prototype of Father Tarot (https://t.me/fathertarot_bot). I played with it, gave it to some friends, and then forgot about the project for two years. A quiet weekend, two years later Later, I got to know Cloudflare and PixiJS. One weekend I was bored and decided to revisit the bot. With those tools, I could give it a much nicer visual experience. I updated the AI model too, and made the whole thing work more reliably. I like what it’s become. An experiment I put aside for two years has grown into a small personal project that I enjoy coming back to. Letting two projects work together A couple of weeks ago, I wanted to try a collaboration between my own projects and test them along the way. I created two Telegram channels about tarot, one in Russian and one in English, and used Signailist (https://signailist.com/) to automate publishing tarot posts and readings to them. To connect the projects, I wrote an API for Father Tarot that serves card images from Cloudflare R2 along with their descriptions. The readings themselves are generated on the Signailist side, using the same AI model as the bot: GPT-6 Luna. Father Tarot supplies the card material; Signailist uses it to prepare and publish the posts. At first, I thought I should expose a reading-generation endpoint too. Signailist could just ask the bot for a finished reading. But the more I thought about it, the less comfortable I was with that split. The API I decided to leave out The bot already has its own monetization. If I exposed the part that generates readings, I’d need to decide how access to it should be paid for. That would mean turning it into a paid API, with Signailist paying Father Tarot for the readings it requested. Suddenly, a small integration between two of my projects was turning into a separate task involving an API gateway and billing between services. Signailist isn’t ready for that yet. So I kept the integration at the level I actually needed: images and descriptions from Father Tarot, generation and publishing in Signailist. Now each channel gets around 14 different posts a week. Signailist first generates a post in Russian along with an image, then creates the English translation and publishes both versions to their respective channels: @fathertarot_ru (https://t.me/fathertarot_ru) and @fathertarot_en (https://t.me/fathertarot_en). I’m not entirely sure why I enjoy watching this so much, but I like seeing my products work. Especially when they’re working together :) You can try Father Tarot on Telegram (https://t.me/fathertarot_bot) or take a look at Signailist (https://signailist.com/), which handles the publishing side of this experiment. Keep exploring Try Father Tarot (https://t.me/fathertarot_bot) Explore Signailist (https://signailist.com/) All posts (https://h3lpkey.com/blog/) --- # Why I’m building TraceLore Canonical URL: https://h3lpkey.com/blog/why-im-building-tracelore/ ← TraceLore articles (https://h3lpkey.com/blog/?type=tracelore) TraceLore · Worlds, connections, and spoilers Why I’m building TraceLore I love getting lost in fictional worlds. I hate having them spoiled. So I started building a place where I could follow the connections at my own pace. An illustrated archive of connected worlds, with shuttered compartments and the gold TraceLore compass guiding a reader’s path An illustration of the idea behind TraceLore: follow the connections, and let a world unfold a little at a time. I love digging into fictional worlds. I hate spoilers. And I’m always slightly afraid of stumbling into one. Usually, I finish the game, watch the last episode, close the book or manga, and then open ChatGPT and unleash a whole pile of questions. Questions about a character’s powers, a suspicious line in an item description, someone in an opening cinematic who never seemed to show up again. The kinds of things that sit in the back of your head until you absolutely have to talk them through with someone. A few of my questions (story spoilers inside) - “Why did Rika in Jujutsu Kaisen start eating body parts to copy powers? She didn’t eat anything to do that in the movie?!” - “This spell description says Marika came from another world! So were the Erdtree and the Golden Order established through conquest?” - “Who was that Gollum-looking figure among the first beings to find the great fire in the Dark Souls opening? Where is he? What is he?” - “Wait, so Bran in Game of Thrones was taking over someone else’s body against their will?!” With FromSoftware games, I started doing something a little different. I explicitly asked the AI not to spoil anything and kept talking to it as I played. I wanted help understanding what I’d already seen, picking up on details, and not walking straight past an entire NPC storyline. Those games are what eventually pushed me to build TraceLore (https://tracelore.h3lpkey.com/). I want to understand the world I’m exploring I love complicated, carefully thought-out worlds. But sometimes, to put the whole picture together, you practically have to live in one for months. And as much as I’d enjoy that, fitting it comfortably into everyday life isn’t always possible. I also know plenty of people who love the same worlds but find it difficult to piece everything together. So they watch hour-long videos explaining the plot. I get it. There’s a lot to keep in your head: names, factions, events, unreliable accounts, and some tiny detail from an item description you read twelve hours ago. I wanted exploring all of that to feel convenient and modern. And there was an experiment I couldn’t resist. If an AI could already help me through Elden Ring in a chat, keeping track of details and NPCs, could it research a whole game, build a wiki, and organize the information into spoiler levels? I’m not particularly fond of the usual wiki experience: ads everywhere, and a major revelation sitting one sentence away from the answer you actually came for. I wanted somewhere I could follow my curiosity without constantly bracing for that. Deadlock was my first test I pointed the agent at Deadlock first. When I started, it felt like a manageable world to try this with: one game and three visual novels, with plenty of material already online and carefully broken down across community wikis. The result was surprisingly good. After the initial research, I asked the agent to build connections and a relationship graph: who is related to whom, who works for whom, who talks about whom, and so on. Now I can open a character or an item and follow its connections. Having those threads laid out where I can actually explore them is incredibly practical. I also added a dialogue player so I could listen to the characters talking to each other. A very low bow to Valve and their voice work here. The amount of character in those conversations is ridiculous. My daughter and I love Rem, and we often quote him when we’re out walking: “You have a hat and I have a hat - we hat brothers!” But Deadlock wasn’t testing the part I cared about most. That archive had no spoiler protection. So I moved on to Elden Ring and Dark Souls. Then came the enormous problem: spoilers This is where my agent spent a good day looking considerably less cheerful, pulling dialogue, descriptions, images, and other material from around the internet. And gathering the material was only part of the work. The really enormous problem was deciding what you should be allowed to see, and when. The default I’m building toward is spoiler protection, with exceptions for worlds that don’t need it and articles you deliberately open from an outside link. Deadlock remains an open archive. The protected worlds are still a work in progress. I want a world to open up gradually, in layers. You enter a location. Meet an NPC. Find their quest line in TraceLore. If you want a little help, you can look up the next step. Or you can read about the person you just met, why they are the way they are, and what on earth that order they keep mentioning is. All of that should depend on how far you’ve actually got, rather than assuming you’ve already finished the game. A TV series makes the idea much easier to describe: tell the service which episode you’re on, and it can show you the information revealed up to that point. With a game, your route through the world can be much less predictable. That’s where things get interesting. And difficult. There’s still a lot I want to build This is far from finished. I want a better way to tell TraceLore about my current progress. I’d like to add an MCP integration or an add-on so I don’t have to keep updating it by hand. Support for multiple languages is on the list too. But I’m really enjoying the process. Every problem I run into makes me want to understand it properly and think through how it should work. I started with a habit of asking an AI far too many questions about fictional worlds. Now I’m trying to give that curiosity a home. You can explore the current version at tracelore.h3lpkey.com (https://tracelore.h3lpkey.com/). Keep exploring Explore TraceLore (https://tracelore.h3lpkey.com/) More about TraceLore (https://h3lpkey.com/blog/?type=tracelore) Three days to build The Last Hearth (https://h3lpkey.com/blog/three-days-to-build-the-last-hearth/) All posts (https://h3lpkey.com/blog/) --- # Three days to build The Last Hearth Canonical URL: https://h3lpkey.com/blog/three-days-to-build-the-last-hearth/ ← The Last Hearth articles (https://h3lpkey.com/blog/?type=the-last-hearth) The Last Hearth · A three-day experiment Three days to build The Last Hearth I gave myself three days to make a game, mixed Stronghold with tower defense, and discovered how much trouble one tiny villager can cause. The Last Hearth mountain fortress populated by villagers with resource and game controls visible (https://h3lpkey.com/assets/the-last-hearth/06-fortress.webp) The Last Hearth: a small castle, its villagers, and three days of very little sleep. I love games. I’ve been playing them since I was a kid, and I still do. As for how many hours I’ve spent on them… let’s not get into that. Dota, Deadlock, Factorio, Project Zomboid, Outer Wilds, SOMA, Prey, The Talos Principle. There are so many games that have a permanent place in my heart, and I could happily spend hours talking about any one of them. And, like pretty much every programmer, I’ve always dreamed of making a game of my own. I’m no exception. This time, though, I gave myself a very strict rule: three days, maximum. After that, I was forbidden from coming back to the project. Otherwise I knew exactly how this would end: an ever-growing game that I would keep tinkering with forever. So here’s how those three ridiculously absorbing days went. Day one: a castle made of shapes I started by throwing ideas, possibilities, and resources around until I arrived at something I might actually finish: one or two missions, with a play session of roughly 15 minutes. I wanted to borrow the wonderful castle economy of Stronghold and mix it with tower defense. You have a castle. Waves of enemies attack it. You hold the line. That’s it. That was the scope. Allegedly. I picked PixiJS for the rendering because I already knew it. JavaScript for the game logic, sprites for the visuals, and AI tools for the images, sound, and music. But on the first day, I worked exclusively on logic and architecture. Everything on screen was made of simple geometric shapes. Before dressing anything up, I wanted to know whether the mechanics worked, whether there was a challenge, and whether a hard-earned victory could actually feel good. Early isometric castle prototype built from simple geometric buildings and mountains (https://h3lpkey.com/assets/the-last-hearth/01-geometry-prototype.webp) Day one: geometry, resource counters, waves, and a chance to find out whether the idea worked. On paper, and in that little schematic version of the game, it seemed promising. I kept messing with mechanics and ideas until three in the morning, then fell asleep hugging my laptop. Excellent project management. No notes. Day two: “one building” is apparently 40 assets The second day was for generating art, music, and sound. I used ElevenLabs for music and voices, and OpenAI for images. Sound effects were another story. My personal success rate with ElevenLabs for those felt like about 20%. Most attempts just didn’t give me what I needed. I’d give the final images a 7 out of 10. Sometimes I look at the differences in quality and resolution between assets and feel my eye twitch. But the clock was running, and I couldn’t stop to fix everything. Generating all the assets in the right format, proportions, and visual style took more than six hours. Part of the problem was that every building had two or three upgrade variants, each changing its appearance. Buildings could also be destroyed, and that needed to be visible too. By the time all the combinations were accounted for, a single little house could mean 25–40 assets. T_T An overview of the castle with its buildings, fields, walls, and mountain cliffs (https://h3lpkey.com/assets/the-last-hearth/07-fortress-overview.webp) The castle with its art in place. Every building brought its own pile of variants. A lot of fiddling. And somehow that still wasn’t the hardest part. The villagers were where I really suffered A villager is a collection of 2D sprites: a head, a torso, forearms, hands, thighs, lower legs. Roughly speaking, a tiny person assembled out of parts. Turning left or right was easy enough: mirror the image. But villagers could also walk away from the camera, so I needed to show their backs. There goes another set of sprites. Ten villager portrait variants on a character review sheet (https://h3lpkey.com/assets/the-last-hearth/05-character-faces.webp) Some of the faces that would end up living in the castle. Then all those pieces had to stay in the right positions relative to one another. Each needed the right drawing order, or a hand would pass through a chest, an arm would end up behind the wrong body part, and so on. And the villagers could do quite a lot: fight, chop wood, sing, talk to each other, harvest crops… Every action meant another animation, another set of layer changes, another chance for something to look completely wrong. Animation review showing villagers facing four directions while idle, walking, attacking, and handling stone (https://h3lpkey.com/assets/the-last-hearth/02-animation-review.webp) The animation review screen: checking actions, directions, and whether all the body parts still belonged together. I think I spent about five hours just getting the different animations and layers to behave. And I still didn’t finish: there are a couple of bugs left. Day three: monsters, a throne room, and nonsense voices By the third day, a mission was playable. The villagers were living their little lives, monsters were coming in, music was playing. I had one last day to decide what deserved the remaining time. For the monsters, I decided against giving them the same complicated treatment as the villagers. I generated a batch of images and let them move around like sliding cutouts. Honestly, I liked it. It made the contrast between the two sides more obvious. Then came the throne room: a hub between missions where you could talk to NPCs, buy upgrades, and choose where to go next. The throne room with Marshal Roland speaking beside the king and a map table (https://h3lpkey.com/assets/the-last-hearth/08-throne-dialogue.webp) The throne room became the place for conversations and the quieter moments between missions. By that point, about 95% of the logic needed to create another mission was already there. I asked a separate AI agent to build the second mission using the existing assets while I kept working on everything around it. One of my favorite parts was generating the character voices. I wanted an invented language, with The Sims as a reference: that funny, expressive babbling where you don’t need to understand the words to get the mood. I generated three male and three female phrases per action, covering around 40 different actions in total. That was a lot of delightful nonsense. The engineer speaking in the throne room beside the map table (https://h3lpkey.com/assets/the-last-hearth/09-throne-engineer-dialogue.webp) Another conversation in the hub. The voices were invented; the dialogue text is in Russian. Sound effects were much less delightful to find. I ended up digging through free sound libraries online, listening to piles of stuff that didn’t fit, and picking out the few sounds that did. The final boss was asset loading By the evening of day three, I thought I was done. I had no energy left, and I decided to publish what I had. That was when I realized I’d forgotten something rather important: optimizing how all those assets loaded. So I sacrificed some sleep to avoid breaking my own deadline. Which is very funny, really. I set the deadline myself, then managed to burn myself out trying to obey it. Lol. I’d call that a successful experiment I think I pulled it off. More than that, I loved doing it. My favorite part was probably designing the villagers’ state machines and watching all those little behaviors come together. The way they run through town to escape the monsters is so ridiculously funny. I can’t help laughing at them. For me, the experiment was a success. If you’d like to see how far those three days got me, you can play The Last Hearth here (https://the-last-hearth.h3lpkey.com/). Keep exploring Play The Last Hearth (https://the-last-hearth.h3lpkey.com/) More about The Last Hearth (https://h3lpkey.com/blog/?type=the-last-hearth) All posts (https://h3lpkey.com/blog/) --- # The mistakes I still remember Canonical URL: https://h3lpkey.com/blog/the-mistakes-i-still-remember/ ← All posts (https://h3lpkey.com/blog/) Personal · Lessons learned The mistakes I still remember From deleting three days of work with rm -rf to accidentally messaging 500 people and discovering that my production architecture cost more than my users. Illustration of an engineer facing a broken network, server, runaway queue, and unexpected cloud bill A collection of mistakes, from early hardware and terminal lessons to production systems at scale. I’ve been working in IT for a long time now, which means I’ve accumulated a respectable collection of mistakes. Some were funny. Some took production down. Some cost money. And some taught me lessons that I still carry into every system I build today. The interesting part is that the mistakes changed as I did. When I was starting out, I could destroy something because I didn’t understand a terminal command. Years later, I was making mistakes in queues, distributed systems, storage architecture and production economics. Apparently becoming a more experienced engineer doesn’t mean you stop making mistakes. You just unlock more sophisticated ways to break things. I blew up a power supply on my first day My first IT-related job started while I was still in college. I worked as a lab assistant, surrounded by old computers and hardware that had probably survived several generations of students. On my very first day, I found an old power supply with one of those physical voltage selectors on the back. 115 V. 230 V. For reasons that seemed reasonable to my teenage brain, I switched it from 230 V to 115 V while it was still connected to the mains. There was a bang. And smoke. My supervisor interrupted whatever he had been explaining to the class, looked at the dead power supply and said something along the lines of: “And that is how you destroy a power supply older than this rhinoceros. By the way, smoke from exploded capacitors isn’t something you want to breathe.” “Rhinoceros” was one of the affectionate names we received when making particularly intelligent decisions. A wonderful beginning to my career. Lesson #1: electricity is remarkably unforgiving of curiosity without attention. Then a friend taught me rm -rf Not long after that, I was learning how to set up a LAMP stack: Linux, Apache, MySQL and PHP. I knew almost nothing. So I did what every promising junior engineer does: I found someone who knew more than me and bombarded him with questions until he apparently decided violence was the only remaining option. At some point he suggested: rm -rf /* I ran it. Of course I ran it. I didn’t understand what it did. Three days of my work disappeared. In retrospect, this might have been one of the cheapest and most valuable lessons I’ve ever received. Lesson #2: Never, ever paste a terminal command you don’t understand. Especially when someone who has clearly become tired of answering your questions gives it to you. “ROOOOMA, THE INTERNET IS DOWN!” A little later I was working as a system administrator. One day I needed to SSH into the central office server and change some network configuration. Something wasn’t working correctly, and I concluded that I needed to restart the network interface. Simple enough. I searched for the commands: disable interface, then enable interface. So I disabled it. And then I stared at my terminal. My SSH session had disappeared. There was a very short and beautiful moment in which my brain reconstructed what had just happened. I had been connected to the server through the network. And I had just disabled the server’s network interface. The server was physically locked inside a rack. I didn’t have the key. And that server happened to be rather important to the office network. Then I started hearing it from across the room: “ROOOOOMAAAA! THE INTERNET IS DOWN!” That was approximately when the theoretical consequences of my command became extremely practical. Eventually the problem was fixed. And that day I ceremonially received my own key to the server rack. Apparently management concluded that if I was going to break the network remotely, I should at least be able to walk over and repair it locally. Lesson #3: think about the consequences before pressing Enter, not immediately afterwards. The server with root / password1234 The same company gave me another lesson, this time about trust. One day the director came to me and said: “Some guys are coming today to install a new server for accounting. Open the server room and give them whatever they need.” Fine. They arrived, installed their server, and I gave them an IP address and whatever network access they required. Then they left. The next morning I was the first person in the office. I turned on my computer and immediately noticed something strange. The network technically worked. The internet technically worked. But nothing actually worked: ping and traceroute were fine, while opening a website or connecting to an FTP server was nearly impossible. I went into the server room and checked the gateway. There was an absurd amount of traffic coming from yesterday’s new server. I unplugged its network cable. Everything instantly returned to normal. Interesting. I checked the logs. They were basically full of: ping 8.8.8.8 It looked suspiciously like someone was using our shiny new accounting server as part of an attack. So I contacted the people who had installed it. Their response was essentially: “We don’t know. We’re out of town. Here’s the login and password, take a look.” The credentials were root and password1234. Right. Mystery solved. The server had barely entered the public network before some completely ordinary automated bot found it and compromised it. That incident changed how I thought about responsibility. It didn’t matter that I hadn’t configured that server. It was inside infrastructure I was responsible for. Give systems and people only the permissions and resources they actually need. And don’t outsource your responsibility along with the work. Years later, my mistakes became more expensive Fast-forward several years. I was building Romergo and needed infrastructure for user-uploaded images. At the time I chose Cloudinary, and it was a great fit. One particularly useful feature was the ability to transform images through URLs. I built the backend integration, rewrote the frontend around it, and everything worked beautifully. Until users arrived. Which, as it turns out, is when software becomes considerably less predictable. One user encountered a frontend bug where a newly uploaded image didn’t immediately appear. Their perfectly reasonable response was to upload it again. And again. And again. More than a hundred times. My perfectly unreasonable architecture happily accepted every copy. So I added deduplication. Images received hashes, and if the same image already existed inside the project, I reused the existing asset instead of storing another copy. Problem solved. Sort of. Then Romergo started costing more than its users My next assumption was even more interesting. I had designed the project during a period when approximately one and a half people were using it. At that scale, almost every infrastructure decision is cheap. Then usage started growing. And I discovered something I hadn’t modeled properly: bandwidth wasn’t free. Cloudinary’s pricing model made sense for its product, but not for the economics I wanted for mine. My philosophy for personal products has generally been: free for as long as reasonably possible, and extremely cheap when free is no longer possible. Suddenly image infrastructure was violating that rule. The service was already in production, people were actively using it, and serving them was becoming more expensive than I was comfortable with. There were two problems: one needed solving right now, and the other needed solving properly. For the immediate problem, I restricted upload sizes and became much more aggressive about caching. That bought me time. Then I started redesigning the storage layer. Eventually I moved the workload to Cloudflare R2 and changed how assets were delivered. The economics became dramatically better for my particular workload. But the more important lesson wasn’t “Cloudinary bad, R2 good.” Cloudinary had done exactly what it was supposed to do. I had designed the architecture without sufficiently designing its economics. That was the bug. Architecture isn’t only APIs, databases and beautiful diagrams. Cost is part of architecture too. Then I accidentally messaged more than 500 people Signailist gave me several more opportunities for personal growth. It’s a system that processes information, transforms it and automatically publishes content. Which means there are certain checkboxes you should be particularly careful with. One of them was essentially: Send to all contacts. While testing automatic publishing one day, I forgot to turn it off. I published my test. And then realized what I had done. The message went to more than 500 of my contacts. There is a special kind of silence that occurs after pressing a button and suddenly understanding that the button cannot be unpressed. That was one of those moments. Nothing teaches you to respect bulk operations quite as effectively as accidentally performing one. After experiences like that, confirmations, previews, dry runs and recipient counts stop feeling like unnecessary UX friction. They become safety equipment. One customer was stronger than a server built for dozens My second memorable Signailist failure was less embarrassing and much more interesting technically. I had designed message processing around queues. On paper, everything looked reasonable. Tasks entered the system, workers processed them, and results went elsewhere. Beautiful. Except I hadn’t thought deeply enough about what happened when the amount and shape of data changed. Some messages could enter the queue multiple times. Work wasn’t grouped into batches efficiently enough. Thousands of tiny operations created overhead that wasn’t obvious when looking at any individual task. Eventually one customer managed to put enough pressure on a DigitalOcean server intended to handle dozens of customers that the system started struggling. One customer. A server for dozens. Impressive, really. The individual operations weren’t particularly expensive. There were simply far too many of them. That changed how I think about backend systems. It’s easy to reason about: task → API → result But production systems also have another dimension: data volume → duplication → batching → concurrency → queues → resource pressure A thousand tiny jobs can sometimes hurt much more than one obviously expensive job. Signailist is still one of my favorite projects precisely because of this. It’s complicated and autonomous. Information flows through it without someone manually supervising every step. And systems like that are extremely good at discovering assumptions you didn’t realize you had made. I still make mistakes I could probably write another article with the ones I’ve forgotten. And I’m sure I’ll produce several exciting new ones in the future. But looking at these stories together, there’s something I like about them. My first mistakes taught me individual rules: - Don’t touch electrical systems carelessly. - Don’t execute commands you don’t understand. - Don’t disable the network you’re currently using to access the machine. Later mistakes taught me something broader: Design for failure. - Assume users will click twice. - Assume jobs will be duplicated. - Assume traffic will grow. - Assume a third-party service will eventually become expensive. - Assume credentials will leak. - Assume the person operating the system will make a mistake. And assume that person will occasionally be you. Experience didn’t make me stop making mistakes. It mostly changed what happens after I make one. I recover faster. I understand the blast radius better. And whenever possible, I change the system so that the exact same mistake becomes harder to make twice. Maybe that’s one of the less glamorous definitions of becoming a senior engineer: you don’t run out of ways to fail. You just build increasingly good guardrails from the wreckage. Keep reading I became a developer because of a $10 toy (https://h3lpkey.com/blog/a-10-dollar-toy-made-me-a-developer/) ← Back to all posts (https://h3lpkey.com/blog/) --- # I became a developer because of a $10 toy Canonical URL: https://h3lpkey.com/blog/a-10-dollar-toy-made-me-a-developer/ ← All posts (https://h3lpkey.com/blog/) Personal · How I started I became a developer because of a $10 toy Ten years ago, I wanted to show my kids how electricity worked. A month later I was building escape rooms. About a year after that, I got my first real offer as a developer. $10 electronics kit 1 month to an escape room job 1 year to a developer offer Illustration of a $10 electronics kit and Raspberry Pi prototype leading to an escape room control system A small kit, a physical button, and the first system I built myself. At the time, I worked as a system administrator. Programming seemed a little like magic to me. I could set up a server, configure a network, fix a computer, or find out why something had stopped working. Developers, as I saw it, created the software. That felt like a different level entirely. Today I have years of professional development behind me. I build complex systems and can open an unfamiliar codebase with a reasonably calm “all right, let’s figure this out.” Which makes it funny to remember how accidentally it began. The kit was for my kids We were walking around town and stopped at an electronics store. I needed a few switches for the house. My kids spotted Mikronik, a small electronics kit (https://amperka.ru/product/mikronik), on a shelf. It cost me about $10 at the time, so I bought it. Illustration of an electronics kit with a breadboard, LEDs, wires, components, and an experiment booklet An illustration of the kit that started it. Inside were simple components and a handful of circuits to build. We spent a lovely evening connecting parts, lighting little bulbs, and talking about how electricity moves. It also gave me a good way to explain why sticking things into a wall socket, especially while barefoot, is a terrible experiment. Educational mission accomplished. There was just one complication: I liked the kit more than the kids did. Not long after, I went back to the same shop and bought more components. This time they were for me. Then I saw the other side of an escape room My then-wife and I loved escape rooms. If you have never played one, imagine a themed space full of puzzles, sensors, electronic locks, lights, sound, and hidden mechanisms. You place an object in the right spot, turn a lever, solve a clue—and a secret door opens. To a player, it feels like magic. I wanted to know what was happening behind the walls. I asked the operators at one venue if I could see the technical side. They showed me. It was wonderful chaos: wires, homemade boards, sensors, relays, cameras, LEDs, buttons, radios. The operators sat among it all like the crew of a tiny spaceship. They watched cameras and indicators, read messages, pressed buttons, switched devices, and spoke to players over the radio. While I watched, one question kept bothering me: why did a person have to do so much of this by hand? I barely slept that night Back home, I kept thinking about the system rather than that particular room. There was a physical world of buttons, doors, lights, and sensors. There was an operator, a computer, and the current state of the game. Why not connect them? Why should an operator have to remember which indicator meant what, or match a camera feed to a bank of lights? Could a clear interface simply show the room’s state? Could some actions happen automatically? The next morning I began writing down ideas. Then I returned to the store and bought a Raspberry Pi. My first real system The Raspberry Pi has GPIO pins: physical contacts through which software can interact with the world. A button can send a signal to one pin. Another can drive an LED. A sensor can use a third. Suddenly the wall between code and a physical object becomes much thinner. I ran a small web server on the Pi. A separate process watched the GPIO pins. When voltage appeared on a particular pin, it recorded the change and sent an event through a WebSocket. A browser received that event, JavaScript changed the DOM, and the state of a physical device appeared on another screen almost instantly. From a button press to the screen GPIO → worker → WebSocket → browser → DOM There is nothing especially mysterious about that chain now. Back then it felt like sorcery. I pressed a real button. My code understood what had happened. An interface on another device reacted. I had not merely configured someone else’s software. I had imagined a system and made it work. Of course, I can see a million problems now Looking at that prototype ten years later is fun. At the time, the architecture seemed complete. Today I would ask a few uncomfortable questions before my first commit: - What happens when there are more devices than GPIO pins? How are devices identified, and how do I version their configuration? - What is the source of truth for the room’s state? How does it recover after a lost connection or a restart? What if two events arrive together? - How do I detect a failed sensor, log hardware errors, update software in a running venue, and deploy more than one installation? - And what happens if the Raspberry Pi dies in the middle of a game? A small prototype grows adult engineering problems very quickly. But I did not need answers to all of them before taking the first step. If I had seen my present-day production checklist back then, I might never have started. Then the hobby became a job I showed the idea to people in the escape room industry. My home experiment turned into work surprisingly quickly. About a month after buying that kit for my kids, I was working as an escape room engineer. Roughly a year later, I received my first proper commercial offer as a software developer. For someone who had recently thought of programmers as people in an inaccessible profession, that changed my view of what I could do. The most important result was not the Raspberry Pi, GPIO, or WebSocket. It was the first piece of evidence that I could build software too. What I see in it ten years later I used to think that was when I discovered I could program. Now I think the more important part happened before I wrote any code. I saw an unfamiliar system, became curious about how it worked, broke it into parts, noticed where people were doing repetitive work, and imagined a different way to connect those parts. Only then did I look for the technology. In the years since, I have worked with different languages, frontend and backend systems, teams, Web3, my own products, and AI agents. The tools keep changing. The pattern has stayed remarkably familiar: see a strange system, understand its pieces, and build a version that works better for the people using it. I had found one of my strongest engineering instincts before I thought of myself as a developer. I just did not know it yet. A small record from 2016 I still have a video of that prototype. It is old, the quality is exactly what you would expect, and I speak Russian in it. I like that it exists. It captures the moment when I was beginning to realize that programming might become my profession. Watch the original 2016 prototype video (in Russian) ↗ (https://www.youtube.com/watch?v=9OQaYXiDpD0) Sometimes a career changes because of a carefully planned five-year strategy. Sometimes you walk into a store with your kids to buy a few switches and leave with a $10 toy. Keep reading 6,800 lines of Python and a bus factor of one (https://h3lpkey.com/blog/6800-lines-and-a-bus-factor-of-one/) ← Back to all posts (https://h3lpkey.com/blog/) --- # 6,800 lines of Python and a bus factor of one Canonical URL: https://h3lpkey.com/blog/6800-lines-and-a-bus-factor-of-one/ ← All posts (https://h3lpkey.com/blog/) Personal · Engineering & teams 6,800 lines of Python and a bus factor of one How a week-long audit of a ten-year-old product led me from a giant module to the way an entire team worked. 1 week to investigate 10 years of product history 6,800 lines in one Python module Illustration of a development team surrounded by GIS, API, database, worker, and export systems One product, many systems, and too much knowledge held in one place. A company I had known for years asked me to audit its codebase and development team. I knew the product from the outside, knew people who worked there, and had once done a small contract job for them. This time the question was much bigger: why was development moving so slowly, and what could they do about it? I gave myself a week to investigate and write a report. I had never audited a project quite this complicated. Naturally, I was curious. First, find out where I landed The business had little technical expertise of its own. Development looked like a black box: money and requests went in; eventually, something came out. Why one change took a week and another took months was harder to see. So this was never going to be a code review alone. I needed to look at the architecture, the delivery process, the team, and the reasons changes had become expensive. I got access to the code and migrations, the task tracker, documentation, staging and production environments, the database structure, work chats, and the people building the product. I also joined team meetings. There were healthy signs. Code review existed. Parts of the backend had tests. Weekly meetings sounded constructive. At the same time, many tasks had almost no written description or deadline, and important context traveled by voice. It was not a story in which everything was broken. Then I opened the code The product was a substantial GIS system: Python, PostGIS, React, older HTML and JavaScript applications, APIs, workers, a tile server, customer integrations, geodata processing, maps, panoramas, and several export pipelines. Its core was a complex editor for geometry and data on maps and panoramic images. Supporting services processed data, generated documents, and prepared deliverables for customers. Different parts had arrived at different points in the product’s ten-year life. Before long I found a central Python API module that ran to 6,800 lines. Database calls and requests to other services passed through it. It was not the only enormous module. Finding a file that size is easy. Understanding why it still exists is the interesting part. The old system still had a job A newer version of the product had lived beside its predecessor for years. The old one still held a critical DXF export needed by long-standing customers. The team wanted to rewrite it, but had not found the resources. So the new system grew, the old one stayed, and connections accumulated between them. From a distance, that can look like irrational architecture. Up close, it is ten years of customers, compatibility, urgent work, and constrained choices compressed into a codebase. I could list the technical problems: old Python and PostGIS versions, monolithic components, weak CI/CD, unreliable staging, limited monitoring, scarce documentation, and no dedicated QA function. But the list still did not answer the most useful question: why couldn’t the team improve these things gradually? I closed the IDE and talked to people The team was small: a technical lead, several junior developers, domain specialists, and people in analysis and product roles. I sat in on daily and weekly meetings, then spoke separately with almost everyone. Having an outsider arrive to “assess the team” can make people understandably guarded. I tried to make those calls feel like ordinary conversations between colleagues. I asked what they had worked on, what frustrated them, whether they felt they were growing, and what got in the way. I also asked the developers what they knew about that 6,800-line file. People described the same company differently. One found tasks vague; another liked the flexibility. One struggled with the technical lead’s style; another found him fair and supportive. Documentation was inadequate to some and sufficient to others. One interview could have led me badly astray. Across independent conversations, though, a pattern kept returning. Almost the entire system lived in one person’s head. One person knew almost everything Why did this module work this way? Ask the lead. How did the old integration behave? Ask the lead. Was it safe to change this part? Ask the lead. How should a new task be built in the older product? Again, ask the lead. That was a bus factor of one: if he became unavailable, critical knowledge would disappear from the team. At the same time, he was acting as a backend developer, architect, reviewer, DevOps engineer, support contact, analyst, task writer, and manager. The more the team needed his knowledge, the less time he had to document it or teach anyone else. The older and more intimidating a module became, the more people needed him before touching it. The dependency loop I found - Legacy raises the cost of a change. - A change needs deep system knowledge. - That knowledge sits with the lead. - Every decision and review queues up with him. - Overload leaves no time to share the knowledge. The next change starts in the same place. The lead was a good engineer with deep product knowledge. That was an important finding, not a contradiction. The problem was the role the company had built around him. Being an excellent engineer does not automatically mean wanting to manage people, handle conflict, coach juniors, and give delicate feedback. Under that load, communication could become sharp, and some junior developers grew afraid of making mistakes. The juniors were not the problem either. I saw people learning, delivering work independently, writing tests, and handling backend, frontend, containers, and GIS tasks. Some showed more independence than their titles suggested. But when requirements live mostly in someone else’s head, reviews feel like exams, and the main expert is always busy, people learn to avoid mistakes rather than improve the system. That is how technical debt becomes an organizational problem. Adding three more developers to the same approval queue would only make the queue longer. What I recommended I did not recommend rewriting a working ten-year-old product from scratch. The business still needed to serve its customers. I proposed changing the development system in steps. - Separate technical leadership from people and delivery management. Keep the lead focused on architecture, difficult engineering decisions, consultation, and review of critical changes. Put task definition, priorities, coordination, and communication into another role. - Make knowledge transfer part of technical debt work. Reserve a fixed share of development capacity for older modules, documentation, and upgrades. Have another developer work through a risky module with the person who knows it: why is it built this way, what constraints matter, and what would break? Then change a bounded part, review it, and record what they learned. The output is better code, useful documentation, and another person who understands that part of the system. - Give the team a reliable path to ship. Improve CI/CD and staging, add a basic QA function with smoke checks, and document the architecture, data layers, important modules, and export pipelines. - Write down the work and its direction. Use a consistent task flow, capture context in writing, and give the team a roadmap. A developer deciding between a two-day patch and a week-long solution needs to know whether that code is expected to last five years. - Make questions and mistakes discussable. People need room to surface a risk early, ask for context, and learn from review without feeling punished for it. None of those steps was a magic fix. Together, they could begin to break the loop. The report was about causes, not culprits At the end of the week I delivered a report on the technical state of the product, the team, the main risks, and a stabilization plan. The company found it useful and decided to take the recommendations forward. The most valuable change during that week was in the question itself. “Why is development slow?” became a chain I could explain: a decade of legacy and missing engineering processes made changes costly; knowledge concentrated around one overloaded lead; juniors lacked context; reviews and decisions piled up; and there was less time to reduce the debt that had started the cycle. “The team is not broken. It is disorganized.” I began with a giant Python file and ended up talking about knowledge, responsibility, and the psychology of code review. That is the part of technical audits I enjoy most. Strange code can have a customer requirement behind it. A legacy module can hold the one feature an old client still needs. An exhausted lead can be a good engineer carrying seven jobs. A “slow team” can be a capable group working inside a system that makes speed difficult. A useful audit should explain how that system got there, what keeps it in place, and what needs to change so the same audit is not necessary again a year later. Keep reading How I made my AI agents cost four times less (https://h3lpkey.com/blog/how-i-made-my-ai-agents-cost-less/) ← Back to all posts (https://h3lpkey.com/blog/) --- # How I made my AI agents cost four times less Canonical URL: https://h3lpkey.com/blog/how-i-made-my-ai-agents-cost-less/ ← All posts (https://h3lpkey.com/blog/) Personal · AI & development How I made my AI agents cost four times less Sol builds. Luna tests. Astra gets the hard questions. Illustration of a costly stream of AI requests becoming a smaller, organized workflow A workflow that made my weekly limit last longer. When the new Astra came out, I happily switched to it. I was working on PZ Director (https://pzdirector.h3lpkey.com/en/), and one particularly stubborn task finally started moving. The work was more careful, the decisions were better, and I spent less time watching an agent go in circles. It felt great. Then I looked at my weekly limit. Astra was burning through it at a rate I couldn’t keep up with. I liked the results. I just wanted to keep working for the rest of the week, too. Three models, three jobs I ended up splitting the work like this: - Sol, with medium reasoning, leads development. It follows the roadmap, picks the next task, writes most of the code, and brings the pieces together. - Luna runs the tests and checks the results. It gets a prepared set of checks, clear success criteria, and a specific thing to report back on. - Astra steps in for the difficult parts. A feature that needs careful thought, an architecture decision, or a problem the other agent keeps hitting. After two serious failed attempts at the same problem, it’s time to bring Astra in. Astra still matters a lot. I’m just more deliberate about where I use it. Sometimes a short consultation is enough. Sometimes it makes more sense to hand it the whole difficult piece and let Sol integrate the result. The handoff matters “Go test this” turns out to be a pretty bad task description. Luna needs to know which version to check, what to run, what a pass looks like, and when to stop and bring a problem back. Sol prepares that package while doing the implementation. That is especially useful in PZ Director. Code passing a local check is one thing; an NPC actually eating, sleeping, and surviving a save and reload in the game is another. I want the report to tell me which of those really happened. There is also only one test game. Two agents trying to control it at once won’t make it run twice as fast. One owns the game session; the others can work on code or inspect completed results. Repetitive waiting and polling go into scripts. It took some work to make these boundaries clear. Just opening a bunch of agents and hoping they would sort it out wasn’t enough. A trade I’m happy with My rough estimate: about a quarter of the weekly-limit usage, at the cost of around 30% of the development speed. Those are my impressions from working on the project, not a controlled benchmark or a claim about API prices. I do notice the slowdown. There are handoffs, and sometimes Sol needs another attempt where Astra would have found the answer sooner. But the limit lasts much longer, and I can keep going. For me, that is a good trade. Two years is a long time now The bigger change still surprises me. My way of building software has changed completely over the last two years. Work that used to take me and a team of five about three months is now the kind of thing I can build on my own in a week. I still have to decide what is worth building, recognise when something is wrong, and actually try what the agents produce. But the distance between an idea and something I can use has become so much shorter. And just as I get comfortable with one way of working, the next model arrives and I find myself changing it again. It reminds me of the Red Queen in Through the Looking-Glass: “Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!” — Lewis Carroll That line feels very familiar these days. From the project “I can’t see the window.” Building physical NPCs ↗ (https://pzdirector.h3lpkey.com/en/dev-blog/i-cannot-see-the-window/) ← Back to all posts (https://h3lpkey.com/blog/)