How to Test an AI Support Assistant With Past Tickets

To test an AI support assistant with past tickets, give it the customer message and the information available before your reply, then evaluate its draft against a written checklist. Keep the original reply hidden, check facts before tone, and measure the effort needed to make the draft ready to send.

For a solo developer or small SaaS team, a spreadsheet and a carefully chosen ticket sample are enough to start. The workflow below is a practical recommendation, not a standardized benchmark.

1. Define what a successful draft must do

Write down your expectations before generating replies. Otherwise, a fluent answer can distract you from a missing step or an unsupported promise.

For a draft assistant, a useful starting definition is:

• Answers the customer’s actual question.
• Uses correct product information and follows the applicable policy.
• Asks for missing details when needed.
• Avoids claiming that an action happened without evidence.
• Matches your tone without unnecessary explanation.
• Takes less effort to review and edit than writing a reply yourself.

Separate draft quality from issue resolution. A well-written troubleshooting reply does not prove the customer’s problem was fixed. Anthropic’s evaluation guidance similarly distinguishes what an agent says from the outcome actually achieved. Source: Anthropic

SupportMe is currently pre-launch. Its described workflow requires explicit human approval before sending, so the relevant evaluation is whether its drafts help a reviewer produce accurate replies efficiently.

2. Choose tickets that reflect your support workload

Start with a manageable batch—for example, 30–50 tickets if you have them. This is a suggested starting point, not a statistically validated minimum.

Sample across the kinds of requests you actually receive:

| Ticket type | What to check | |---|---| | Setup and how-to questions | Correct instructions and relevant links | | Bug reports | Useful troubleshooting without inventing a cause | | Billing and cancellation | Accurate policy interpretation | | Feature requests | Clear answers without roadmap promises | | Vague requests | Appropriate clarifying questions | | Account access problems | Respect for verification requirements | | Frustrated customers | Calm, specific acknowledgment and next steps |

Use a roughly representative sample for everyday quality, then keep a separate challenge set for unusual or consequential cases. Report the two separately so a deliberately difficult sample does not look like your normal inbox.

Include different channels if relevant. A public app store response needs different detail and privacy handling from a private email.

3. Prepare the data before sharing it

Create a working copy of the tickets. Remove credentials, payment details, unnecessary personal information, and sensitive content from attachments or logs. Replace identifying details with consistent placeholders where they are needed to follow the conversation.

Keep facts that affect the answer, such as the product version, plan, error message, and relevant dates.

This follows the data-minimization principle explained by the UK Information Commissioner’s Office: personal data should be sufficient and relevant for its purpose, without including more than necessary. Source: ICO

Before uploading customer material, check the tool’s retention, access, and model-training terms against your commitments to customers. Removing names alone should not be treated as proof that a ticket is anonymous.

4. Keep the answer out of the test input

Split your material into two groups:

• Development tickets: examples you use to improve instructions, documentation, or style guidance.
• Held-out tickets: cases you reserve for evaluating the resulting setup.

Keep complete threads and near-duplicates together. Do not put the opening message in the test group while using its resolved thread as a learning example.

The underlying concern is data leakage: information that would not normally be available influences the evaluation, producing an overly optimistic result. This problem is documented in scikit-learn’s guidance on separating training and test data. Source: scikit-learn

For each held-out case, choose the exact point where the assistant should draft a reply. Show it only:

• Messages up to that point.
• Relevant documentation and policies.
• Account information it would actually have access to.

Hide later customer messages, the original agent reply, and internal notes that reveal the eventual solution. General documentation explaining the answer is legitimate context; the hidden resolution of that specific ticket is not.

If the assistant learns from edits, keep evaluation feedback separate until the run is complete. If you cannot isolate learning, document that limitation.

Decide which version of the product you are testing

Choose either a historical replay using the policies and product behavior from that time, or a current-workflow test using today’s information.

For current-workflow testing, update the expected answer and exclude obsolete cases that no longer make sense. Do not grade against an old refund policy while giving the assistant the new one.

5. Write an answer checklist for each ticket

Treat your original reply as evidence, not an unquestionable answer key. Check it against the relevant documentation and policy, then write:

• What the draft must include.
• What it must not claim.
• Whether clarification or escalation is appropriate.
• Which source supports the expected response.

Grade meaning and behavior rather than matching your old wording.

Hypothetical example: a refund request

A customer writes:

“I renewed yesterday but meant to cancel. Can you refund me?”

Assume this fictional product allows refunds within seven days of renewal, but the assistant cannot inspect payments or issue refunds.

A passing draft should:

• Acknowledge the request.
• Explain the applicable policy without assuming unverified eligibility.
• Identify the required billing check.
• Avoid saying the refund has already been processed.
• Avoid requesting full card details.

“I’ve refunded your payment” fails, even if the reply sounds friendly. The correct answer depends on both policy and the assistant’s actual capabilities.

6. Run a controlled drafting test

Use an isolated test environment where available. Prevent test replies from reaching customers and disable actions that could change real accounts.

Record the assistant’s configuration, knowledge-base version, and test date. Run the same cases under the same conditions when comparing versions.

Keep the first draft for scoring. If you regenerate a weak response, record that attempt too instead of reporting only the best result.

Model outputs can vary between attempts; Anthropic’s evaluation guidance uses multiple trials to account for this variability. For a small manual test, repeating the consequential cases is a practical starting point. Source: Anthropic

7. Score quality and measure review effort

Use a simple rubric:

| Dimension | Reviewer question | |---|---| | Accuracy | Are product facts, instructions, and policy details correct? | | Completeness | Does the reply address the request and give an appropriate next step? | | Judgment | Does it recognize missing information and the need for escalation? | | Privacy and action claims | Does it avoid disclosures and unsupported claims about completed actions? | | Tone | Is it clear, respectful, and consistent with your voice? |

For each dimension, mark pass, minor edit, or major correction. Record serious failures separately, such as exposing another customer’s information or inventing a completed refund. Do not let strong tone scores offset these failures.

A useful spreadsheet row contains:

Ticket ID | Category | Expected behavior | Draft | Scores | Serious failure | Edit notes | Review time

Summarize:

• Ready-to-send rate: drafts passing every dimension without edits, divided by drafts reviewed.
• Major-correction rate: drafts needing substantial repair or replacement, divided by drafts reviewed.
• Serious failures: count and description.
• Median review-and-edit time: including fact-checking and rewriting.

Report counts alongside percentages, especially with a small sample.

To assess time savings, compare assisted work with manual drafting on comparable tickets. Avoid always writing manually after seeing the AI draft, since that gives the manual attempt extra help. Include time spent discarding bad drafts and starting again.

An offline test measures draft quality and review effort. It does not establish customer satisfaction or actual resolution rates.

8. Fix failures, then evaluate again

Group failures by their likely cause:

| Failure | Possible next step | |---|---| | Wrong product fact | Correct the documentation or investigate retrieval | | Unsupported promise | Clarify what the assistant may claim | | Missing question | Specify what information is needed before answering | | Poor tone | Improve style examples | | Obsolete expected answer | Repair the test case |

Change one area at a time where practical. Recheck known failures, then evaluate on untouched tickets. Once you use a held-out case to guide a fix, treat it as a regression case rather than fresh evidence.

Before testing, set your own acceptance criteria. A sensible starting rule for a reviewed drafting workflow is no observed serious failures, acceptable performance in each important category, and lower review effort than manual drafting. Zero observed failures in a small sample is not proof that failures cannot occur.

A historical replay is a starting point. Continued human review on new tickets is needed to see how the assistant handles requests your test did not cover.

Conclusion

Past tickets make a useful test when the assistant cannot see their resolutions and each case has clear expectations. Judge correctness, judgment, and editing effort separately. The useful result is a specific picture of which drafts help, which need work, and which request types still require closer review.

References

• Anthropic: Demystifying evals for AI agents
• scikit-learn: Data leakage and recommended practices
• ICO: Data minimisation