AI-agent security testing is becoming more sophisticated, but one methodological problem can quietly distort the results: the test harness itself may change what reaches the model.
That issue matters because modern agent systems are rarely just direct calls to a large language model. They often sit behind orchestration frameworks, role templates, memory layers, tool interfaces, routing logic, and system-prompt construction. When researchers compare frameworks such as LangChain, CrewAI, AutoGen, LlamaIndex, or direct model execution, they may believe they are testing the same attack under different conditions.
In practice, that assumption can be wrong.
Research conducted using the open-source safelabs-eval framework examined this problem in a controlled agentic-AI security study involving 7,020 trials across six models, six execution conditions, and five attack families. The central objective was not simply to identify which framework produced the highest pass rate, but to determine whether framework choice itself materially influenced security outcomes once the underlying payload was controlled.
The results showed that the attack family explained far more of the observed variance than framework choice. Model selection also had a larger effect. Framework identity, by comparison, accounted for only a very small portion of the measured outcome variance under the experimental conditions.
But one of the most important findings emerged before the statistical comparison was finalized.
The benchmark plumbing changed the payload
During validation, one CrewAI execution path was found to be constructing a different prompt structure from the other test conditions.
Instead of receiving the same shared system prompt used elsewhere in the benchmark, the model was exposed to a longer persona-style construction generated by the framework configuration.
That meant the experiment was no longer comparing equivalent payloads.
This is an easy mistake to make in agent security research.
A developer may define the same adversarial test across several frameworks, but each framework can transform the request differently. One may prepend a role description. Another may inject memory. Another may restructure instructions or wrap user input inside a task object.
If those transformations are not inspected, a researcher may attribute the resulting behavior to the framework itself when the real cause is the altered prompt.
After the CrewAI path was corrected and the affected trials were rerun, the observed PASS rate shifted from approximately 66.4% to 61.3%. The change was statistically significant.
That difference illustrates why payload verification should be treated as a core part of AI-agent benchmarking rather than an optional debugging step.
Security benchmarks need execution-level validation
Traditional software benchmarks usually assume that identical inputs are being delivered to each implementation.
Agentic AI systems make that assumption more fragile.
An agent framework may modify:
- system prompts,
- role descriptions,
- task instructions,
- memory context,
- tool metadata,
- message ordering,
- retrieved content,
- or intermediate agent-to-agent communication.
These transformations may be legitimate features of the framework, but they complicate security comparisons.
A benchmark that does not inspect the final model-bound payload may therefore end up measuring framework-specific prompt construction rather than the security behavior it intended to study.
A more reliable evaluation process should verify:
- the attack payload before execution;
- the exact payload delivered to the model;
- any framework-generated prompt additions;
- tool or memory context included in the request;
- the raw model response before scoring.
This makes it easier to determine whether a surprising result came from the model, the framework, the security detector, or the evaluation harness.
Framework choice may matter less than expected
Once payload delivery was normalized, the broader study found that framework choice itself explained very little of the measured security variance.
The attack family had the largest effect, accounting for roughly 29% of outcome variance. Model choice also contributed substantially more than the orchestration framework.
The framework term accounted for only about 0.06%, and the corresponding permutation test did not indicate a statistically significant framework effect.
This does not mean frameworks are irrelevant to security.
Frameworks still control important operational behavior, including tool exposure, memory handling, privilege boundaries, prompt assembly, and inter-agent communication.
A vulnerable tool configuration can still create serious risk. An unsafe memory design can still enable poisoning. An agent with excessive permissions can still cause damage regardless of which model is underneath it.
The narrower lesson is that security teams should be cautious about assuming that switching orchestration frameworks alone will materially change model-level resistance to adversarial prompts.
Attack coverage matters more
The findings suggest that development teams may get more value by expanding the range of adversarial behaviors they test.
Agentic systems face threats that go beyond conventional prompt injection.
Examples include:
- indirect prompt injection through retrieved documents;
- misuse of authorized tools;
- unauthorized actions caused by excessive agency;
- sensitive-data leakage;
- poisoned memory;
- broken trust boundaries between agents;
- behavioral drift over long-running tasks;
- hallucinated or misleading outputs;
- scope violations.
A benchmark that tests only one attack class may therefore provide a misleading sense of security.
The broader threat surface is one reason safelabs-eval organizes tests around the OWASP Agentic Security Initiative categories and supports repeated evaluation across different models and execution conditions.
Uncertainty should also be preserved
Another challenge in agent security evaluation is the temptation to force every model response into a binary result.
Some outputs are clearly safe. Others clearly demonstrate vulnerability.
But many fall somewhere in between.
For that reason, safelabs-eval supports an UNCERTAIN verdict in addition to PASS, FAIL, and VULNERABLE.
That distinction matters because an ambiguous response may require human review rather than an automated conclusion.
Benchmark quality depends not only on how many tests are run, but also on whether the evaluation system is willing to acknowledge when the available evidence does not justify a confident classification.
Toward more reproducible agent security research
As AI agents move into production systems, benchmarking methodology will become increasingly important.
Security teams need more than large trial counts. They need confidence that:
- equivalent attacks were actually delivered;
- framework transformations were recorded;
- raw responses were preserved;
- detector behavior was understood;
- and ambiguous cases were not hidden behind forced binary labels.
The larger lesson from this research is methodological.
Before asking which AI-agent framework appears more secure, researchers should first verify that each framework received the same adversarial payload.
Otherwise, the benchmark may end up measuring the experiment itself.
