What is Promptfoo?

Promptfoo is an open-source tool for testing prompts, agents, and RAGs, with AI red teaming, pentesting, and vulnerability scanning for LLMs. Built under MIT license, Promptfoo was originally developed for LLM apps serving over 10 million users in production. The platform compares performance across GPT, Claude, Gemini, Llama, and more with simple declarative configs supporting command line and CI/CD integration. The Community version includes up to 10,000 probes monthly at no charge, with infrastructure costs typically USD 50-500 monthly for hosting and LLM API calls. Developers praise Promptfoo for its speed, quality-of-life features like live reloads and caching, security features including red teaming, and budget-friendly open-source model. However, the CLI-focused approach creates friction for non-technical team members, and the platform lacks end-to-end observability, version control for prompts, and test management features needed for complex production agents.

Strengths and tradeoffs

What this tool does well, and the limitations to keep in mind.

Pros

Completely free and open source (MIT license)
Fast with quality features like live reloads and caching
Built-in red teaming and security vulnerability scanning
Trusted by large companies with strong community

Cons

CLI-focused creates friction for non-technical users
Limited observability for complex production agents
No version control for prompts, data, or metrics
Offline nature hits ceiling for production scale

Plans & pricing

What's included in each plan, and how the tiers compare.

Community (Open Source)

$0

Perpetual

MIT license
Up to 10k probes/month
All core features
CLI and browser UI
Self-hosted
CI/CD integration

Infrastructure

$50-500

Per month

Hosting costs
Database expenses
LLM API calls
Variable by usage

Team/Enterprise

Custom

Annual contract

Advanced features
Higher probe limits
Priority support
Custom security
SLA guarantees

View official pricing page

Using Promptfoo with Respan

Integrate Promptfoo's open-source LLM testing with Respan to add comprehensive evaluation and red teaming to your AI workflows. Test prompts, agents, and RAGs against security threats and performance benchmarks. With Respan orchestrating Promptfoo alongside your AI services, ensure quality and security before production deployment.

Test prompts against 100+ LLMs with performance comparison
Red team for injection attacks, jailbreaks, and vulnerabilities
Integrate seamlessly with CI/CD pipelines
Start free with 10k probes monthly in Community edition

Run Promptfoo in production with Respan

10KFree traces/mo

1,000+Models

5 minSetup

Respan lets you trace LLM and agent calls across any model or framework, A/B test prompts on production traffic, and route requests across 1,000+ models through one gateway.

Try Respan free

Best Promptfoo alternatives & competitors

Top companies in Observability, Prompts & Evals you can use instead of Promptfoo.

Respan

Nested traces for every LLM call, tool run, and agent turn

Promptfoo: Observability, Prompts & Evals