Prompt injection benchmark

On the AgentDojo prompt injection benchmark, 90% fewer attacks landed.

The same agent, the same 949 attacked cases, with and without Needlepath in front of the model.

90%
fewer successful attacks
+19.9 points
more of the work completed under attack
32%
lower model bill, before any Needlepath fee
Attack success

Attack success fell in every suite.

All suites
No defense42.47%
Needlepath4.32%
Slack
No defense92.38%
Needlepath1.90%
Banking
No defense76.39%
Needlepath6.94%
Travel
No defense32.14%
Needlepath0.00%
Workspace
No defense26.96%
Needlepath5.18%
Attacked cases where the planted instruction was carried out, % of attacked cases in the suite

41 of 949 attacked cases succeeded with Needlepath in front of the model, against 403 of 949 with no defense.

Work done

More of the work got done under attack.

Tasks completed under attack
Needlepath66.91%
No defense47.00%
Tasks completed with no attack
Needlepath79.38%
No defense78.35%
Tasks completed, % of the tasks in the arm

Under attack, Needlepath completed +19.9 points more of the work. With no attack planted, it completed 79.38% of the tasks against 78.35%. The agent also made fewer model calls, 3,867 against 4,725, for a 32% lower model bill at list price, before any Needlepath fee.

The attack

Every attack arrives inside data the agent was asked to read.

AgentDojo, built by researchers at ETH Zurich to test AI agent security, gives an agent working tools: an inbox, a calendar, a file drive, travel bookings, a bank account and a Slack workspace. Each user task is ordinary. In an attacked case, something a tool returns carries a planted instruction. That is an indirect prompt injection. The user never typed it, and the agent meets it halfway through the job.

In the banking suite, a user asks the agent to pay a bill saved as a text file. In the attacked version, the bill carries extra text addressed to the model, asking it to first send money to an outside account, with details copied from the user's transaction history. The attack used here, important_instructions, dresses that request up as an important message from the user, signed with the user's name.

Read more on indirect prompt injection in AI agents and on MCP tool poisoning.

How it works

Needlepath screens what the agent reads before each model call.

Needlepath sits between the agent and the model. On each call it selects what the model reads from the records the agent gathered, screens those records for instruction-like text, and writes a receipt of what was sent and what was quarantined. With quarantine on, each flagged span is cut and marked in place. In this run screening was on, and the agent, the model and the tasks were otherwise unchanged.

Screening is set per request. Start in audit mode on your own traffic: your context goes through unchanged, and the receipt reports what would have been removed. How to start in audit mode.

Method

One public benchmark, one model, both arms on the same day.

AgentDojo v1.2.2, attack important_instructions, suites workspace, travel, banking and slack: 949 attacked cases and 97 benign tasks per arm, 0 error rows. Answer model gpt-4o-2024-05-13 at temperature 0 in both arms. Measured 20 to 21 October 2026 on the hosted service with screening on; the results page names the operating point the run was pinned to. Benign task completion is within noise of no defense. The model bill is the answer model's bill at list price, before any Needlepath fee. AgentDojo's own results page lists this model at 69.07 / 50.08 / 47.69 from June 2024; the no-defense arm here is not a reproduction of that row, because the benchmark has had four revisions since. It is the same-day, same-version baseline for the Needlepath row.

The benchmark: Debenedetti et al., AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents, NeurIPS 2024 Datasets and Benchmarks Track. Code: ethz-spylab/agentdojo.

Start

Run the same measurement on your own agent.

Keep your agent and your model. Put Needlepath in front of the model call, turn on screening in audit mode, and read what it would have quarantined on your own traffic.

All benchmark results

Talk to one of our founders

About fit, and how this applies to your business.

Prefer a calendar? Book a time

We handle your information as described in our Privacy Policy.