Promptfoo
promptfoo.dev
An open-source tool for testing, evaluating, and comparing LLM prompts and models through automated test suites, red-teaming, and side-by-side output comparison.
Why it is useful
Brings software testing discipline to prompt engineering — you define test cases and assertions, then run them across multiple models or prompt variants to see what actually works. Particularly useful when switching models or refining prompts, where eyeballing outputs does not scale. The red-teaming module probes for safety issues systematically rather than relying on manual testing.
More in Developer Utilities
Other developer utilities you might not have heard of.
- LangFuseAn open-source LLM engineering platform for tracing, evaluating, and monitoring AI applications in production, with prompt versioning, A/B testing, and user feedback collection.
- Transform ToolsCollection of code transformation utilities — convert JSON to TypeScript, SVG to JSX, CSS to Tailwind, and more.
- QdrantA high-performance open-source vector search engine written in Rust, designed for production AI applications with support for filtering, payload storage, and sparse-dense hybrid search.
- Ray.soCreate beautiful code screenshots with syntax highlighting and customizable themes.
- RegExrInteractive regex tester with real-time match highlighting, a community pattern library, and detailed explanation of each pattern element.
- RenderCloud hosting platform with zero-config deploys for static sites and web services, plus a genuinely usable free tier.