All stories
Agent Evidence Vectors Ecosystem Community reports

Can your agent prove it finished? Probity offers 2,736 tests

A fresh Hugging Face snapshot puts 57 corpora of agent-evidence checks within reach. They test verifiers against known cases, rather than certifying that an agent is safe or successful.

By AITrending2 min read
Sources (5)

Why now November 3 mirror commits identify the v0.32.1 source snapshot, following a November 2 source tag. The Hugging Face project itself dates to October 24; this is a new snapshot, not the project's first launch.
Probity's terminal demonstration shows verified, failed and not-established verdicts for intact, altered and missing evidence, followed by a changed-verifier comparison.
Final frame of the maintainer's four-arm terminal demonstration in the v0.32.1 source README. These are author-recorded demonstration outcomes, not an AITrending test run or proof of universal agent reliability. Source: Probity AI / Agent Evidence Vectors contributors

An agent saying “done” is a claim. Probity’s Agent Evidence Vectors gives verifier builders a collection of records with known expected outcomes to check what their software actually accepts. A November 3 Hugging Face mirror identifies the project’s v0.32.1 source snapshot and provides 2,736 cases across 57 corpora. AITrending counted those rows and distinct corpora in the reviewed snapshot; it did not execute the verifier suite.

Despite the name, these are conformance test cases, not embedding vectors. Each case identifies a corpus, a file path and associated manifest information. Expected outcomes are either carried in the dataset row or retained inside the referenced vector file. The intended audience is developers building or evaluating checks for agent execution evidence, including in-toto attestations, completion receipts, audit records, MCP and A2A records, and SCITT statements.

Test the evidence behind the claim

The tagged repository describes cases involving altered artifacts, missing evidence, approvals bound to the wrong action, fabricated tool output and completion claims without a matching confirmation. Its author-published terminal demonstration contrasts an intact archive with a changed byte, a missing artifact and a changed verifier. That recording illustrates the maintainer’s workflow; it is not an independent AITrending reproduction.

For a team integrating agents into real work, the useful question is concrete: does its verifier reject a record when the evidence no longer supports the claim? The repository includes a harness for calling an external verifier and retaining a conformance report. Passing a set of fixtures does not establish that an agent completed arbitrary real tasks, eliminate every attack or certify regulatory compliance.

Pin the snapshot, and check the package version

The dataset is Apache-2.0 and exposes separate cases and corpora configurations through Hugging Face Datasets. Its card also documents downloading the repository and checking SHA256SUMS before evaluating the saved files. A reproducible review should record an immutable revision, such as the reviewed Hugging Face commit de628fcc8afba92864c2f00e353e97c74cd6d431, alongside the source identity and corpus digests.

There is a material documentation mismatch. The Hugging Face card calls the snapshot v0.32.1 and suggests matching package and Action pins. However, the source distribution guide at that tag still identifies v0.32.0 as the published release and describes package metadata and criterion-document repairs as unreleased. The downloadable snapshot is observable; readiness of every advertised installation route is not established by that observation.

The Hugging Face project was created on October 24. The new hook is its November 3 mirror, not a claim that all 2,736 cases were invented today or that the project has suddenly gone viral. For verifier builders, it is a specific collection worth inspecting against their own evidence contracts and threat model.

Related reading