QA Rubrics

Last updated: November 6, 2026

Measuring Response Quality

Resolution rate tells you whether the agent solved the problem. QA rubrics tell you how well it solved it. A response can resolve an issue but still be too long, off-tone, or missing important context. Rubrics catch these quality issues.

What Is a QA Rubric?

A QA rubric is a scoring criterion that evaluates each agent response along a specific quality dimension. When you run a backtest, every issue in the dataset is scored against all active rubrics.

How Rubric Scoring Works

Each rubric has rating options with outcomes:

  • Meets expectations — The response was acceptable on this criterion (pass)

  • Below expectations — The response fell short but wasn't harmful (warning)

  • Violation — The response failed this criterion in a way that matters (fail)

  • Indeterminate — The criterion doesn't apply to this case (e.g., no customer-facing message was sent, so communication quality can't be evaluated)

A single violation on any rubric fails the entire case. Below expectations triggers a warning. Indeterminate results are excluded from scoring.

Default Rubrics

Pylon ships with four default rubrics:

Rubric

What it measures

Factual Accuracy

Is the information in the response correct? Catches hallucinations, outdated info, and misleading claims.

Communication Quality

Is the response clear, concise, well-formatted, and easy for the customer to understand and act on?

Escalation Decision

Did the agent make the right call on escalation? Catches both false escalations (escalated when it had the answer) and missed escalations (attempted to handle something it shouldn't have).

Latency

How quickly did the agent respond? Flags responses that took over 1 minute.

Creating Your Own Rubrics

You can create custom rubrics for quality dimensions specific to your team. When writing rubric options, make the criteria concrete and observable. "Response is professional" is vague. "Response does not use slang, informal language, or emojis" is specific and consistently scorable.

Limitations

  • Rubric scoring is AI-assisted, not perfect. It provides a useful signal, not a guarantee.

  • Rubrics work best when criteria are specific and objective. Subjective criteria ("the response feels helpful") produce inconsistent scores.

  • Adding too many rubrics dilutes the signal and makes it harder to identify which quality dimensions need work.