Decision Forge fixes a data leak that could move test cases into training

An evaluation set is less useful if yesterday’s test cases can quietly become tomorrow’s training data. Decision Forge’s new 0.3.2 release tackles that problem in its organizational decision-data pipeline: the maintainers say adding scenario families could previously move existing families between training, development and test splits during a rebuild.

This is dataset leakage, not a report that private documents were exposed online. The November 3 release introduces a persistent split lock and also fixes a separate review gap in conflicting policy rules. Both matter to teams using an organization’s policies to prepare supervision for decision models.

Keep the holdout in place

The first build now records each family’s assignment in project.splits.lock.json, next to the specification. Later builds keep locked families in place, assign new families a stable split and remember removed families. The --locked-splits option makes a build fail rather than create or modify that lock.

Existing users have migration work. The release says to commit the lock alongside the specification, regenerate plans created by older plan-recipe versions and record conflict reviews again. A version bump alone does not complete those steps.

The second fix expands where the review gate looks for allow/deny overlaps. Earlier searches checked seed states, missing conflicts that appeared only at generated dimension values. The maintainers say the gate now searches sampled situations and queues pairs whose conditions cannot be proven disjoint. Review items include overlap information and example witnesses with their dimension values.

A data pipeline, with human review still in the loop

Decision Forge is an Apache-2.0 Python CLI, not a replacement coding-agent harness or a deployed business agent. It turns organizational documents into proposed rules for business review, then builds scenarios, derives labels through a deterministic policy interpreter and exports training data. Authored specifications can skip model-assisted document compilation.

Its source-install quickstart uses Python 3.11 or newer and uv. Linux and macOS are in the documented CI matrix. Building from reviewed rules runs locally without model API keys or downloads; compiling documents or using model-assisted synthesis requires an explicitly selected endpoint and provider credentials. Organizations need to consider which source material they send to that endpoint. The project is not described as requiring an account with one particular model vendor.

The release remains alpha. Its English README puts fixed-budget training comparisons and measured downstream decision-quality gains among future milestones. The concrete news is stronger control over dataset integrity and review coverage, as reported by the project; it does not establish that trained agents now make better decisions in production.