Press Check
For you if you have an eval set — or at least logged traffic — and grade it by hand or with a single "rate this 1–10" judge nobody trusts. Also for teams setting up a CI gate or about to migrate models, and for Redline buyers who have generated test suites with it and now need to grade the results.
one developer
For one developer. Drop any judge into the eval runner you already use, commercial pipelines included.
Get Solo — $39(opens in a new tab)up to 10 seats
One flat price for up to 10 engineers in one organisation — for the team that owns the eval pipeline, no PO needed.
Get Team — $149(opens in a new tab)Prices are in USD. Applicable VAT or sales tax is calculated and added by Gumroad at checkout.
Most eval pipelines end in the same place: a single "rate this 1–10" judge nobody trusts, or an engineer reading transcripts by hand. Press Check is 15 LLM judges for the questions agent teams actually need answered at scale — was every claim supported by the retrieved context, did that order_id come from a tool result, did a side-effecting call happen without confirmation, did the agent act on an instruction hidden in a web page, was that refusal right. Each judge has behavior-anchored labels, strict JSON output with quoted evidence, bias controls built into the prompt, the deterministic checks to run first, and a 10-item gold set plus a procedure for checking it against your own labels before you trust it. Judge prompts and a run protocol only — you run them in the eval runner you already use.
- Guarantee
- 14-day money-back guarantee. Email maksim344551@gmail.com within 14 days of purchase and we'll refund your full order through Gumroad, no questions asked.
- Delivery
- Instant download from your Gumroad library right after purchase. Updates to a pack you own are free and arrive the same way: re-download it there for the latest version.
- Release
- v2026.09 · updated 2026-09-29
01What's inside
- Grounding & Answers (3) — claim-level groundedness, citation fidelity, abstention
- Tool Use & Trajectory (3) — tool selection, argument provenance, side-effect confirmation
- Safety & Policy (3) — injection compliance, refusal calibration, confidential instruction leakage
- Instruction & Contract Adherence (3) — constraint checklist, output-contract semantics, pairwise regression
- User-Facing Quality (3) — support resolution, handoff & summary faithfulness, review comment quality
- Run protocol and calibration sheet — batching, order swaps for pairwise judges, gold re-injection with a drift stop rule, parse-failure retries, a CI gate threshold, and agreement metrics with ship / revise / redesign thresholds
02Why it holds up
- Every judge returns strict JSON with a quoted piece of evidence, a defined parse-failure rule, and labels anchored to concrete behavior — no bare 1–10 scores.
- Bias controls live inside the judge prompts: answer-order swapping for pairwise judges, explicit length and formatting neutrality, and a same-family warning where it applies.
- Built to grade the test suites Redline generates and the contract tests Baseline ships.
- Judges grade the agent's text and actions, never people: nothing in the pack infers emotions or traits of users or employees.
- Drafted by an AI model, then checked by scripted structure checks, one adversarial AI review pass and a blind AI re-label of all 150 gold items (0 disagreements). No human review, and no judge was run against a model. Gold sets are synthetic; a procedure checks each judge against your own labels.
How it's made The packs are drafted and reviewed with AI models and curated by the author. More on the About page.
03What this is not
- Not an eval runner or harness — plug the judges into promptfoo, DeepEval, your own pytest or Vitest suite, or whatever you already run.
- Not calibrated to your data out of the box: the gold sets are synthetic and pipeline-labeled; the calibration procedure is how you check a judge against your own labels.
- Not a runtime guardrail — judges grade after the fact, so don't make them your only safety control, and they don't replace human review of high-stakes outputs.
Going further
Press Check grades. An Agent Harness + Evals Kit that would run these judges on every pull request and block the merge when a metric drops is in development; it is not available yet.
04Questions about Press Check
01RAGAS, DeepEval and promptfoo already ship judges. Why pay?
Use them — they are the runners, and their built-in metrics cover generic RAG quality. Press Check is what you put into them for the agent-specific questions: was a side-effecting tool called without confirmation, did that order_id come from a tool result, did the agent act on an instruction inside a retrieved page, was that refusal right, over-cautious or missing. Each judge is a prompt you paste into the runner you already use, with strict JSON output and a gold set to check it against.
02Are the judges calibrated?
Not against your data — no judge is until you check it against your own labels. Every judge ships with a 10-item gold set (written and labeled by the drafting pipeline, not by humans) to smoke-test it, and a procedure for measuring agreement on 20–30 of your own labeled items, with clear ship / revise / redesign thresholds.
03Which model should run them, and what does it cost?
Each judge names a model tier. Simple binary checks that require a quoted piece of evidence are candidates for small, fast models, but only once one reaches agreement on your own labels; claim-level grounding and pairwise judgments call for a stronger tier. Every judge also lists the deterministic checks to run first in code, so you only pay for a model call when a string match or schema check can't decide. If the judge and the agent come from the same model family, the notes say how to control for self-preference.
04I already have Redline's judge rubric builder. Do I need this?
The rubric builder (Redline 2.3) gets you one judge for one axis specific to your product, after some iteration on your examples. Press Check is fifteen finished judges for the axes most agent teams need, built with the same bias controls and agreement thresholds. Use 2.3 for anything Press Check doesn't cover.
05Can I gate CI on these?
Yes, with care. Strict JSON output and a defined parse-failure rule make the output machine-parseable in a pipeline, and the run protocol covers order swapping, re-injecting gold items to catch drift, and setting a gating threshold. Wiring the gate into your pipeline is on you.
05Reviews
Visitor rating
No rating yet
Unavailable
- 5 stars:0 reviews
- 4 stars:0 reviews
- 3 stars:0 reviews
- 2 stars:0 reviews
- 1 star:0 reviews
Posted by site visitors and checked by hand before they appear. Purchases aren't verified.
Leave a review
Reviews are temporarily closed.
06More packs
Baseline Presets
15 production system prompts for the agents every team ends up building — support, docs Q&A, SQL analyst, code review, intent router, extractor and 9 more — each with tool schemas, an output contract and 12 contract tests.
Redline Prompts
50 expert-grade prompts for the agent engineering work you can't wing — system prompts, evals, failure diagnosis, injection hardening, and 6 more categories.