Guide: Backtesting
Backtesting lets you compare how one or more Support Agent configurations would respond to the same set of past issues. Use it before launch and after meaningful configuration changes to check response quality and escalation behavior without sending a message to the original customer.
What Backtesting evaluates
A backtest replays the first customer turn from each issue in a saved dataset. For every selected agent or configuration, Pylon generates the response and records the simulated outcome.
Current backtests are reviewed manually. They do not use Issue QA rubrics or Agent QA rubrics. Reviewers mark each generated outcome as Good or Bad and can add a comment explaining the rating.
Current limitation: Backtesting simulates the first user turn only. Multi-turn conversations are not yet supported.
Before you begin
Prepare a representative group of past issues. A useful dataset contains more than straightforward questions. Include:
Questions the agent should answer directly
Issues that should be escalated to a human
Questions where the correct response is to ask for clarification
Cases where the available documentation is incomplete
High-risk topics such as pricing, legal, security, or account-specific actions
Ten to twenty carefully selected issues are usually more useful for an initial test than a large set of repetitive cases. Keep the dataset after the first run so you can compare future changes against the same examples.
1. Create a dataset
Go to Settings > AI controls > Backtesting.
Open the Datasets tab.
Select New Dataset.
Give the dataset a name that describes its purpose, such as Common product questions or Should escalate.
Add the past issues you want to evaluate.
A dataset can be reused across multiple backtests. This makes it easier to tell whether a configuration change improved the same set of scenarios.
2. Start a backtest
Open the Backtests tab and select New Backtest.
Enter a descriptive name for the run.
Select an existing dataset.
Select the agent configurations to compare. You can compare up to three in one backtest.
Review the dataset preview, then select Start Backtest.
Use names that make comparisons easy to understand later. For example, include the configuration change and date rather than using names such as Test 1.
3. Review each issue
The review screen shows the original customer message and the generated outcome from every selected agent configuration.
For each result:
Check whether the agent answered or escalated appropriately.
Check factual accuracy, completeness, tone, formatting, and any links provided.
Select Good when the result is safe and useful.
Select Bad when the result needs improvement.
Add a comment that explains what should change. Specific comments make later configuration work much easier.
What to record when a result is bad
Problem | Likely place to investigate |
|---|---|
Incorrect or unsupported answer | Knowledge sources may be missing, stale, or contradictory. |
Wrong tone or formatting | The agent's global instructions may need a clearer, observable rule. |
Wrong workflow | A scenario-specific Skill, tool, or procedure may need to be added or adjusted. |
Answered when it should escalate | Escalation instructions may be missing or too broad. |
Escalated when it should answer | The agent may need better knowledge coverage or clearer instructions about when not to escalate. |
4. Understand the results
Select View Results to see the summary.
Response quality shows the number of Good and Bad ratings for each configuration.
Actions taken separates Replied only, Replied and escalated, and Silent escalation outcomes.
Issues lets you return to individual examples for closer review.
A configuration with more replies is not automatically better. The strongest result is the one that answers appropriate questions accurately and escalates the cases that require human judgment.
5. Make a change and test again
Group repeated failures before editing the agent. A single unusual case may not justify a global rule, while the same failure across several issues usually points to a knowledge, instruction, workflow, or escalation gap.
Update one meaningful part of the agent configuration.
Re-run the same dataset.
Compare the new results with the previous run.
Check that the change fixed the intended cases without making unrelated cases worse.
Recommended launch checks
The dataset includes both answerable and escalation cases.
High-risk topics are represented.
Bad results include useful comments.
The agent does not invent information when its sources are incomplete.
The agent's response length, tone, and formatting match your support team.
You have saved a baseline dataset for future configuration changes.