Reading Backtest Results

Last updated: November 6, 2026

Understanding What the Numbers Mean

After a backtest completes, you get a set of results for each issue in the dataset. Here's how to read them.

Key Metrics

Resolution Rate

The percentage of issues where the agent's response would have resolved the customer's question. This is your primary metric. A higher resolution rate means the agent is handling more issues without human intervention.

Escalation Rate

The percentage of issues where the agent chose to escalate rather than respond. A high escalation rate isn't always bad — it can mean the agent is correctly identifying questions it can't handle. But if the escalation rate is high on questions the agent should be able to answer, that's a sign of missing KB content or skills.

QA Rubric Scores

If you've configured QA rubrics, each issue is scored against them. Rubric scores tell you about the quality of responses, not just whether they resolved the issue. See the "QA Rubrics" article for details.

How to Compare Runs

The most valuable use of backtesting is comparing two runs: before and after a change. When comparing:

  • Use the same dataset — Comparing results across different datasets is meaningless

  • Look at both improvement and regression — Did the change help on targeted issues? Did it hurt on unrelated ones?

  • Focus on categories — A 5% overall improvement might mask a 30% improvement in one area and a 10% regression in another

What to Look For

Good signals

  • Resolution rate increasing on the targeted question type after a skill or KB change

  • No regression on unrelated question types

  • Escalation rate decreasing on questions the agent should handle

Warning signs

  • Resolution rate improved on one type but dropped on another (regression)

  • Agent is answering questions it used to escalate, but the answers are wrong (false confidence)

  • QA rubric scores dropped even though resolution rate went up (lower quality resolutions)

Drilling Into Individual Issues

Don't just look at aggregate numbers. Click into individual issues, especially:

  • Issues where the agent did worse than expected — what went wrong?

  • Issues where the agent escalated but probably could have answered — what was missing?

  • Issues where the agent answered but the response quality was low — how could the skill or KB be improved?