DEV Community
•
2026-08-03 09:30
Fail the build when your prompt gets dumber: evalgate for prompt regression CI
Prompts rot silently. I swap a model, tweak a system prompt, add a tool, and everything still runs. No exception is thrown, no test goes red, the JSON still parses. The output is just quietly worse, and I usually find out from a user rather than from CI. Unit tests are the wrong instrument here because there is nothing to catch: the failure mode is not a crash, it is a drop in quality.
So I built...