LoginSign Up
← Back to all sparks
P

Promptfoo

AI-ASSISTANTS
Velocity5.0

Open-source LLM evaluation, testing and red-teaming toolkit

www.promptfoo.dev ↗

Promptfoo tracks every frontier model within days, and now ships itself as agent skills

llm-evaluationred-teamingprovidersagent-skillsassertionsdeveloper-tools
◆Current state
Promptfoo is an LLM evaluation and red-teaming harness, releasing at a pace set by the model market rather than by its own roadmap. The provider list absorbs new frontier models almost as they launch — Claude Opus 5, Claude Sonnet 5, Claude Fable and Mythos, GPT-5.6 on Bedrock and at GA, Grok 4.5, Kimi K3, plus Bedrock coverage for GLM, MiniMax, Nemotron, Gemma and Palmyra. Around that, the work is harness ergonomics: per-test repeat, websocket URL templating and subprotocol surfacing, YAML source locations in prompt reference errors, and a steady stream of assertion-scoring corrections.
◆Where it's heading
Two things are happening at once. The provider matrix is a commodity race the project has decided to win on latency-to-support, which makes promptfoo useful precisely because it is never the reason you can't evaluate a new model. The more interesting move is distribution: publishing its four red-team skills to the Claude Code marketplace puts evaluation and adversarial testing inside the coding agent rather than in a separate CLI run. The assertion fixes — BLEU brevity penalty, inverse operators on cost and latency, zero thresholds honoured — suggest the scoring layer is being tightened as people rely on it for gates rather than exploration.
◆Prediction
Expect same-week support for the next frontier model releases to continue, with further red-team capability packaged as agent skills rather than only as CLI commands.

◆Recent moves

  1. 9h ago

    Promptfoo adds Claude Opus 5, Kimi K3 and websocket URL templating

    Claude Opus 5 support lands alongside refreshed Azure, Claude and Gemini model lists and Kimi K3 on the Moonshot provider. Websocket URL templating is the one harness feature; the rest is keeping pace with the model market.

    View source ↗
  2. 16d ago

    Promptfoo adds GPT-5.6, Grok 4.5 and an Open Interpreter provider

    GPT-5.6 on Bedrock and at GA, xAI Grok 4.5, Meta's Model API and an Open Interpreter provider all arrive in one release. The breadth per release is the product's main claim.

    View source ↗
  3. 23d ago

    Promptfoo adds per-test repeat and broad Bedrock model coverage

    A per-test repeat option and YAML source locations in prompt reference errors improve the harness itself, while Bedrock gains Grok 4.3, GLM, MiniMax, Kimi, Nemotron, Gemma and Palmyra. Claude Sonnet 5 lands across four hosting surfaces at once.

    View source ↗
  4. 1mo ago

    Code-scan action fixes mixed skip responses and moves to Node 24

    Two fixes to the GitHub Action wrapper: findings in mixed skip responses are now processed, and the action runs on the Node 24 runtime. Tooling upkeep separate from the main package.

    View source ↗
  5. 1mo ago

    Promptfoo unpins Docker Python and fixes over-redaction

    Docker builds stop breaking on Alpine base-image drift, RAGAS context-relevance segmentation drops stranded list markers, and the sanitizer stops redacting benign URLs that merely mention a credential keyword. Housekeeping across build and scoring.

    View source ↗
  6. 1mo ago

    Promptfoo publishes its red-team skills to the Claude Code marketplace

    ⚡ SPARK

    Alongside Claude Fable and Mythos support, all four promptfoo red-team skills are published to the Claude Code marketplace — the point at which the product stops being only a CLI you run and becomes something an agent invokes. The assertion corrections in the same release point at a scoring layer being trusted for gating.

    View source ↗