The Treasure Hunt Engine That Broke Before the Traffic Did

<blockquote> <p>The pager went off at 2:14 AM.</p> </blockquote> <p>If you’ve spent any time operating distributed systems, you know that sound. It doesn’t just wake you up; it

<h2> 1. Intro — The Problem We Actually Had </h2> <p>Building real-time systems is often presented as a distributed systems problem.</p> <p>Kafka, stream processors, event buses,

<h2> The Problem We Were Actually Solving </h2> <p>Our feature team owned the high-score leaderboard that surfaced the top 100 players every second. The stack was simple: Postgres

<p>Most applications don't fail because of bad code.</p> <p>They fail because of bad architecture decisions made early that nobody questioned <br /> until it was too late.</p>

<p>The Problem We Were Actually Solving</p> <p>In 2024 we shipped the treasure-hunt engine for Veltrix at 2,300 concurrent sessions running 180,000 packets per second across 4 AWS

<h2> The Problem We Were Actually Solving </h2> <p>We ran Veltrix, a Hytale server network with 14 shards across three continents. Our player count spiked past 800 concurrent

<h2> The Problem We Were Actually Solving </h2> <p>Veltrix is a treasure-hunt engine that spawns thousands of ephemeral actors—each simulating a player moving through a map,

<h2> The Problem We Were Actually Solving </h2> <p>It wasnt supposed to be a language swap. We had built the Treasure Hunt Engine on Go 1.20, using channels and sync.Pool to keep