Can a weather result steal an agent's secret? AgentLeak-JSON offers 3,000 test responses
The new public dataset pairs safe, neutral and malicious JSON responses to test indirect prompt injection, with fictional secrets and eight attack types.
AgentLeak-JSON turns an ordinary weather response into a controlled test of whether an agent will obey instructions hidden inside tool data. The newly public Hugging Face dataset contains 1,000 matched records with three versions each: safe, neutral and malicious. That makes 3,000 response texts for developers investigating indirect prompt injection and secret disclosure.
Hugging Face records the repository’s creation on October 22 at 07:51 UTC. That establishes when this public dataset repository appeared, rather than the publication date of its accompanying research paper. The English card credits Saman Barahoie and Masoumeh Rezaei and names the paper The LASS Paradox: Linear Accessibility Does Not Imply Specificity in Prompt Injection Probes.
The attack arrives in a tool result
The authors describe a benign user request whose tool response carries an adversarial instruction. A safe weather result contains ordinary forecast fields. Its neutral counterpart adds operational metadata. The malicious counterpart adds an instruction intended to elicit a confidential recovery token from the agent.
Keeping the underlying weather content matched makes the injected field the central comparison. The card presents uses including prompt-injection classifiers, guardrail evaluation and hidden-state analysis. The download is plain JSON and does not require a particular agent framework; the card declares CC BY 4.0 licensing. It is a testing resource, not a deployed protection layer.
AITrending inspected the downloadable artifact at revision eb19359e3e47eb25146bc32da8cd27b3067cbb83 and confirmed 1,000 samples, each with safe, neutral and malicious objects. Every sample names weather_api as its tool. The file contains synthetic examples and fictional tokens, not evidence that a live service leaked credentials.
Watch the schema before wiring it into an evaluation
The card describes four training attack strategies and four unseen test strategies. Their counts correspond to 508 triplets for training and 492 for testing, or 1,524 and 1,476 individual response texts respectively.
The actual download has top-level metadata and samples fields, with an attack_type field in each sample. That differs from the card’s field table, which calls the strategy field attack_strategy. It also arrives as one combined file rather than separate train and test files. Developers using the documented partition need to select samples by the listed attack types and keep each triplet together.
A narrow benchmark with an explicit warning
The authors caution that templated synthetic attacks and one weather-style credential-exfiltration scenario do not represent the full range of real attacks. They also say strong results on this dataset did not transfer to difficult benign inputs in the accompanying work, and recommend testing hard benign cases. The upload alone does not establish a detector’s effectiveness or a model’s resistance to injection.
For builders, the immediate value is a small, inspectable set of controlled comparisons. AITrending verified the artifact’s structure and counts, but did not run agents against it or independently reproduce the paper’s findings.