GitHub says secret-scanning alerts won 11.5% more acceptance in A/B test

An A/B result for a scanner already in preview

GitHub says repository owners accepted 11.5% more secret-scanning alerts in a customer A/B experiment with its new classifier than with its previous language-model pipeline. The October 13 technical report adds measured customer behavior to GitHub’s October 7 announcement of a ModernBERT-based detector for unstructured credentials. GitHub weights the acceptance rate by repository owner so a customer resolving many alerts at once does not dominate the comparison. Higher acceptance may indicate fewer false positives, but the result is GitHub’s own experiment, not an independent audit of every credential the system might miss.

Faster post-push checks, with a rollout still underway

GitHub describes a two-stage design: regular expressions find possible secrets, then a fine-tuned ModernBERT classifier judges each candidate using its code context. Against the prior pipeline, GitHub reports 13% higher precision and 19% higher recall in an evaluation on the same public traffic. It also reports an 86% reduction in median post-push scan execution time and more than 98% lower serving cost. Those comparisons concern GitHub’s own workloads and baseline. The company says the faster design makes generic secret detection practical during push protection, but its earlier announcement placed that use in private preview with broader access planned later in October. The new report does not establish that the planned wider rollout has finished.