Jev Decision Index fixes a scoring bug affecting 37 models
Version 0.3.2 restores three omitted same-skill test sets and adds a public-versus-private chart to a community leaderboard for typed decision models.
A correction to the comparison
The community-maintained Jev Decision Index says it fixed a scoring bug in version 0.3.2 on October 12. According to its changelog, three same-skill test sets had been left out of the same-skill score for 37 models evaluated since version 0.3 launched. The index now assesses every model against all of those sets. Its maintainers also added a chart showing each model’s public result beside its private same-skill result. This is an update to the evaluation, not the release of a new Jev model.
What the index is measuring
The project tracks open attempts to reproduce TypeSafe AI’s Jev-style typed decisions. Its stated Full score gives 20% weight to public benchmarks, 50% to private tests of the same skills and 30% to private tasks from new domains. That design is meant to make a strong public-only result less decisive. The 0.3.2 fix matters because leaving test sets out of one component can change how models compare under that formula. The omission and correction are documented by the index maintainers; AITrending has not independently rerun the private tests or determined which individual ranks changed.