A Background Job Was Crashing Our JVM Every Week - Until We Taught It to Stop

<blockquote> <p>The pager went off at 2:14 AM.</p> </blockquote> <p>If you’ve spent any time operating distributed systems, you know that sound. It doesn’t just wake you up; it

<p>Two days into a batch classification run, I opened the log and saw line one again. Not a continuation of line one — the same line one, with a new timestamp and a new process

<blockquote> <p>Skip the theory rabbit holes. This is the caching knowledge that shows up in system design interviews, code reviews, and the 2 AM production incidents nobody

<p>The thing that breaks before 1,000 users is almost never the code.</p> <p>Not a race condition. Not a memory leak. Not a missing index. Founders assume their app will collapse

<p>A retry loop is an availability problem before it is a queue-setting problem: it can spend all worker slots on work that cannot succeed and raise the age of healthy messages.

<p>Your AI system isn't going to crash. It's going to drift.</p> <p>A recommendation engine making 1.4 model calls instead of 1. A retrieval pipeline fetching 5 chunks instead of

<p><strong>Part 2 of 4 — Laravel Architecture Patterns for Production</strong><br /> <em>~9 min read · Queue design · Job architecture · Background processing</em></p>

<p>On March 31, 2026, Anthropic accidentally shipped the entire source code of Claude Code — its flagship AI coding agent — inside a routine npm update. No hack. No reverse