A decision model's 0.906 score becomes 0.791 after a clean re-score

The number that changed

Phocinae-Largha-150M-v1’s reported accuracy on the Typed Decisions benchmark was 0.906. After another participant questioned whether its training had used material derived from the test split, a model contributor acknowledged the overlap and reported a clean re-score of 0.791. On October 13, the benchmark maintainer replaced 0.906 with 0.791 in the dataset’s fitted-model table and added a footnote linking the public discussion. The old figure remains part of the record, but it is no longer the table’s ranking value.

Why the comparison matters

The contributor said the training mix had contained 500 error-correction pairs mined from predictions on test rows. For the re-score, the team says it substituted other training items while keeping the broader recipe and the 400-case evaluation the same. Those setup details and both scores are the contributor’s account, not an independent replication by AITrending. The dataset distinguishes models fitted on its training split from general models evaluated without that fitting; the two tables should not be read as one universal ranking. The correction shows how test exposure can make an apparently exceptional score a poor basis for comparing models, even when the model itself remains available.