Backtesting Best Practices

Last updated: October 30, 2026

How a backtest works

For each ticket in the dataset, the backtest:

  1. Takes a snapshot of the original conversation up to the first agent reply.

  2. Spins up a fresh playground session seeded with that snapshot.

  3. Runs your chosen variation end-to-end against the snapshot.

  4. Mocks outbound actions — the agent's "send message," "escalate," and "seek assistance" tool calls are captured but not executed. No customer is ever messaged.

  5. Optionally grades the resulting transcript with rubrics.

Everything else the agent does is real: it queries your knowledge base, it searches feature requests, it reads issue context. Only the outbound customer actions are mocked.

Building a dataset

Datasets are built from a Pylon issue view. Filter your issues however you want — by category, by tag, by date — save it as a view, and convert that view into a backtest dataset.

A few rules:

  • Maximum 100 cases per dataset. If you need more coverage, build multiple datasets per category.

  • 5-15 cases is plenty for iterating quickly. Big datasets are slow to run and produce noise you don't need at the iteration stage. Build one tight dataset per category, run it on every change, and keep a larger validation dataset for milestones.

  • Tag tickets by category so your dataset is intentional. A blend of unrelated tickets makes it harder to read regressions.

First-turn only

Backtests replay the customer's opening message and judge the agent's first response. They do not simulate the customer's follow-up replies. If a ticket was only resolved through a back-and-forth conversation, that resolution won't be reproduced in the backtest.

Pick tickets where the first turn was the decisive moment: a clear question with a clear answer, a clear gap that should escalate, a clear feature request. Avoid tickets where the customer's first message was ambiguous and the agent only succeeded because of a clarification.

Rubrics — structured grading

A rubric is a scoresheet the judge model applies to each backtest case. Each rubric:

  • Has a question (e.g. "Did the agent escalate appropriately?").

  • Has multiple-choice options with point values.

  • Is versioned — every edit creates a new version, and existing backtest results stay tied to the version they ran under.

You can write your own rubrics or use Pylon's standard rubric library as a starting point. Standard rubrics cover the most common dimensions: escalation correctness, factual accuracy, tone, citation discipline, and so on.

Rubrics are applied to the agent's response, not to the ticket overall. They evaluate "how did the agent do" on this case, not "what's the truth about this ticket."

Comparing variations

The most useful backtest workflow is A/B: run the same dataset against two variations and compare. Typical comparisons:

  • Live vs test. Your current production variation against a candidate change. The candidate must clearly win without regressing elsewhere.

  • Old vs new harness. When you migrate from an older agent architecture, compare the new variation against your previous live agent on the same dataset.

When reading results, look for:

  • Per-case wins/losses. Did the candidate fix the specific cases you targeted?

  • Per-rubric scores. Did the candidate improve the dimension you cared about (e.g., escalation correctness) without regressing others (e.g., reply tone)?

  • Surprising regressions. A change that wins overall but breaks an unrelated category is often worth a second look.

When to refresh a dataset

Datasets get stale. Reasons to refresh:

  • Your product changed and the agent's "correct" behavior on old tickets is no longer correct.

  • Your KB changed (articles added, deprecated) and the ground truth has shifted.

  • You expanded to a new ticket category that isn't in your dataset.

A rough cadence: build datasets when you start an iteration cycle, refresh them every 1-2 months as the product evolves.

Cost and concurrency

Backtests run cases in batches of 5 with concurrency 5 — so a 50-case dataset runs roughly 10 batches deep. Each case costs the same as an agent invocation on a real ticket (model + tool calls). Plan accordingly when running large datasets.