Setting Up Your First Backtest

Last updated: November 6, 2026

What Is Backtesting?

A backtest runs your AI agent against a set of real historical issues to see how it would have responded. It's the primary way to measure your agent's performance and validate that changes (skills, KB updates, prompt edits) actually improve outcomes.

Why Backtest?

  • Establish a baseline — Before making changes, know how your agent performs today

  • Validate improvements — After adding a skill or updating KB content, backtest to prove the change helped

  • Catch regressions — Verify that a change to one area didn't break another

  • Build confidence — Data-driven evidence that your agent is ready for broader coverage

Step 1: Create a Dataset

A backtest dataset is a collection of real issues from your support history. Good datasets:

  • Represent your actual volume — Include a mix of question types in roughly the proportions you see in production

  • Include both easy and hard questions — Don't cherry-pick only questions the agent should handle well

  • Start small — 5-15 issues is plenty when you're getting started or iterating on a specific question type. Small datasets run fast and make it easy to read each result individually. As your agent matures, build a larger dataset (30-50 issues) for milestone validation and regression testing.

  • Are recent — Issues from 6+ months ago may not reflect your current product or support patterns

Step 2: Run the Backtest

  1. Navigate to your agent's backtest settings

  2. Select the dataset you created

  3. Choose the agent configuration to test against (your live agent, or a clone with proposed changes)

  4. Start the backtest

Backtests take time proportional to the dataset size. A 50-issue dataset typically completes in 10-20 minutes.

Step 3: Review Results

See the "Reading Backtest Results" article for how to interpret what you get back.

Building Good Datasets Over Time

Start with a small dataset of recent, representative tickets. As you identify specific areas for improvement (e.g., feature request handling), create targeted datasets focused on those question types. Over time, you'll have:

  • A general dataset for regression testing

  • Per-category datasets for targeted skill validation