If you're building an AI agent, you've probably had to choose between frameworks like:
- LangChain.
- CrewAI.
- AutoGen.
- LlamaIndex.
- OpenAI Agents SDK.
- Google ADK.
- Semantic Kernel.
That decision affects architecture, developer experience, integrations, and orchestration.
But does it materially change how secure the resulting agent is?
We ran 9,360 controlled adversarial trials to investigate.
The short version:
Attack category mattered a lot.
Model choice mattered.
Framework choice barely moved the result.
But getting to that conclusion exposed an important benchmarking problem.
Frameworks don't necessarily deliver identical prompts to the model—even when your test harness thinks they do.
We discovered this ourselves when one adapter altered the system-prompt construction enough to measurably change the results.
That led us to enforce payload identity before interpreting framework differences.
We then went beyond conventional significance testing and performed formal equivalence testing across all eight execution conditions.
All 28 pairwise comparisons satisfied our pre-defined equivalence boundary.
There was one small exception worth discussing: CrewAI retained a statistically detectable residual effect, although it remained inside the practical-equivalence threshold.
There were also two implementation surprises involving reasoning-token exhaustion and cross-framework token accounting.
Full technical breakdown:
Open-source evaluation framework:
https://github.com/AgentSafeLabs/safelabs-eval
AgentPort-Bench:
https://github.com/AgentSafeLabs/agentport-bench-results
For developers building production agents, I'm curious:
Do you choose frameworks partly based on perceived security characteristics—or mainly based on engineering and ecosystem considerations?
Top comments (1)
"Framework choice barely moved the result" is a genuinely useful negative finding, and enforcing payload identity before interpreting differences is the methodological discipline most benchmarks of this type skip entirely.
The implication is worth stating plainly, because it points somewhere most agent-security advice doesn't go. If the framework doesn't determine whether the agent gets compromised, then the security can't live in the framework. It has to live in what the agent can reach once it's already compromised.
That's the layer we work at on Krova Cloud : each Cube is a microVM with its own kernel and no public IP, so the reachable set is what you deliberately opened rather than what the harness happened to permit.
Honest scope: your trials measured injection resistance, not containment. A hardware boundary does nothing about an agent being talked into doing something harmful within the permissions it legitimately has. Both layers are needed and they don't substitute.
Did the attack-category effect hold consistently across models, or did some models flatten it?