All stories
SelfBench Ecosystem Community reports

SelfBench makes timed-out coding tests count as failures

Its latest production release fixes DeepSWE execution paths involving E2B, Modal and Codex, while changing how test timeouts enter benchmark results.

By AITrending2 min read
Sources (4)

Why now The production release was published October 17 at 01:23:45 UTC. Its dated notes establish execution and scoring changes rather than a new model leaderboard victory.
SelfBench's author-published interface showing an accuracy-versus-cost chart for a repository benchmark.
Still from the English README's author-published demonstration. Displayed scores, task counts and relative dates belong to that recorded example; they are not current measurements, independent AITrending results or evidence of the timeout-scoring change. Full frame preserved. Source: Mupt AI / SelfBench contributors

Timeouts no longer disappear into the error bucket

SelfBench now counts tests that exceed their time limit as failed trials with reward zero, rather than excluding them as errors. The change appears in its October 17 production release alongside fixes for running Harbor registry benchmarks such as DeepSWE through its supported execution paths.

The release labels those trials “Tests Timed Out” and supplies a backfill tool for older runs. It also says results now default to configurations with the most finished non-error trials. That makes the scoring change consequential: comparisons depend on which attempts count, as well as which model produced a working patch. The notes do not establish that every historical result has already been backfilled or that any particular model’s published score has changed.

Fixing the environment around the agent

The same release says E2B sandboxes preserve environment variables set by task images so verifiers can find tools such as pytest. It fixes Codex installation on images that ship NodeSource Node, and makes Modal pull each task image once before its trials, with retries when a registry throttles requests. E2B and Modal are execution environments; Codex is an agent harness. They are different parts of the evaluation stack, not three competing foundation models.

SelfBench’s broader workflow builds tasks from a repository’s merged pull requests, including hidden tests and a reference solution. Users inspect and approve tasks, select model and harness pairs, run comparisons in fresh sandboxes, then examine transcripts and accuracy-versus-cost results. Its evaluation guide says the tested agent receives the base repository and instruction, not the held-out tests or reference solution.

The web application requires GitHub sign-in and repository access. Workspace administrators configure model and sandbox credentials; account and harness compatibility still matters. For example, the guide restricts a ChatGPT sign-in to Codex evaluations. Provider and sandbox usage remain separate from SelfBench’s MIT source license. Local development requires Bun and Docker Compose, while its documented production reference uses GCP services.

AITrending reviewed the English release and tagged documentation, without reproducing these execution fixes. The news is a more explicit treatment of failed attempts and a set of concrete environment repairs, not proof that generated tasks are representative of every repository or that one agent is universally best.

Related reading