N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

N-Day-Bench arrived the way the best research tends to: no fanfare, just a link dropped into the right pond and left to ripple. The premise is disarmingly simple and long overdue. Take real vulnerabilities from real production codebases — the kind with CVE numbers, public advisories, the whole paper trail — then wind the clock back. Strip out the patches, strip out the context, feed the raw code to a language model, and ask: can you find what went wrong? These are n-day vulnerabilities, meaning the flaw is already known to the security community, which lets researchers score the model's answer against a ground truth that actually exists. No guessing whether the model found something real. Either it spots the vulnerability or it doesn't. That's the elegance of the benchmark design, and it's why the Hacker News crowd gave it enough upvotes to surface. The technical commentary in that thread, as of April 13th, was already probing the methodology — asking how much context the models receive, whether the benchmark controls for code that might have leaked into training data, and what "finding a vulnerability" even means at different levels of specificity. Those are the right questions, and the fact that the community is asking them suggests the research is credible enough to pressure-test.

If confirmed as a rigorous and reproducible benchmark, what this means is a step-change in how we evaluate AI-assisted security tooling. Right now, most claims about LLMs finding vulnerabilities are anecdotal, cherry-picked, or evaluated on synthetic code that no real engineer would write. A standardised benchmark against actual production flaws changes the negotiating terms entirely — security teams can ask vendors for a score, not a story. The second-order effect is sharper: if top models perform well on n-day benchmarks, the pressure mounts to test them on zero-days, which is where the genuinely uncomfortable questions about AI-enabled exploitation begin. For the vibe coding crowd — developers who are shipping faster than ever with AI assistance — a credible vulnerability-detection benchmark also reframes what "good enough" looks like in automated code review. And for anyone building or procuring AI security tools, N-Day-Bench hands them a measuring stick they didn't previously have.

Watch for independent replications of the benchmark results, and for disclosure of how thoroughly the evaluation set was scrubbed against known training data — that methodological detail will determine whether this is a real signal or a well-packaged artifact.

NewsHive monitors these sources continuously. All signal titles above link to the original reporting.

Intelligence by NewsHive. Need help navigating what this means for your business? Contact GeekyBee →