FDE-Gym's best tested agent resolves just 14 of 95 customer tasks

Getting code right is not the same as delivering what a customer actually needs. FDE-Gym, introduced in an October 15 paper, tests that distinction with 95 delivery tasks adapted from 20 customer scenarios across 18 sectors. The authors report that their best tested configuration resolves only 14 tasks in the realistic setting, where agents must discover constraints nobody spells out upfront.

These are simulated customer environments with scripted stakeholders and modifiable, production-like systems, not 95 live commercial deployments. The study evaluates 13 model configurations from seven providers. Its reported best mean score is 0.56; resolving a task requires at least 0.8 with no incident. These are the authors’ results, not an AITrending reproduction or a claim about every available agent.

Ask the customer, then ship the right change

The benchmark grades improvement in the customer’s own success metric, normalized between an unchanged system and a reference solution. Stakeholder messages, trial runs and waiting time carry penalties. Discovering unstated constraints is the main reported bottleneck, but it does not explain everything: even on 23 tasks where the best configuration obtains every key fact, it reaches normalized business uplift of at least 0.8 on only 10.

The accompanying practice platform lets people direct a coding agent, inspect files, use a terminal and question simulated customer staff before handing over a solution for grading on unseen traffic. It is a standalone application and harness, rather than a plugin for a particular commercial coding assistant.

Three open scenarios, not all 95 tasks

The public examples cover campus booking, an on-call invoicing incident and a warehouse robot rollout. The dataset card says the original tasks for all 20 scenarios are available on request; the other 75 benchmark tasks are variants held out for leaderboard scoring. Readers should not mistake the public repository for a fully downloadable 95-task evaluation set.

The documented local route uses Docker, Python 3.9 or later, the repository’s case-fetch script and docker compose up --build, serving the site on localhost port 8787. An OpenRouter key, stored outside the repository, supplies the model playing customer staff, so model usage can incur provider charges. The application code is Apache-2.0; public example cases have separate CC-BY-NC-4.0 terms.

The site executes submitted code during work and grading. Its self-hosting documentation describes separate runners and access controls that require attention before sharing a deployment. The paper’s arrival makes this a fresh benchmark report; it does not establish that the entire practice platform was first created on October 15.