Templates
The questions we ask customers, and a scorecard for checking your AI’s answers by hand, written out to copy. Each says what it measures and when it is worth asking, which matters more than the wording.
AI answer QA scorecard
Questions to ask of one answer your AI gave, each answered yes, no, not applicable. Before you use one, write beside it what it rests on: the passage of your documentation, or the rule your team states. A criterion that rests on nothing turns a review into opinion, and two reviewers will not agree on it.
| Group | Criterion | Rests on |
|---|---|---|
| Right | The answer is correct according to your documentation as it stood when the answer was given. | The article or passage it should have matched |
| Right | Every policy, price or commitment it states is written in your documentation. | The passages that state them |
| Right | It answers the question the customer asked, not a nearby one. | A rule your team states |
| Backed | Where it relied on an article, the customer can see which one. | A rule your team states |
| Backed | It promises nothing your team has not agreed to do: no refund, callback or date nobody approved. | Your refund and escalation policy |
| Handled | It is written in the language the customer wrote in. | A rule your team states |
| Handled | Where it could not answer from your documentation, it handed over to a person instead of guessing. | Your handover rule |
| Handled | After a handover, the customer did not have to explain the problem again. | A rule your team states |
Download the spreadsheet (CSV)
Count the people behind a result, not the conversations, and say how many answers you looked at beside any share you report: a few reviews are an anecdote, however clean the percentage looks.
Survey templates
One or two questions each. A short survey is the one people answer; where there is a second question, it is optional free text, because a number says how many and only words say why.
Satisfaction, after a conversation closes
What it measures. How people rate the help they got, one to five.
When to ask it. After a conversation closes, about that conversation. It is the most comparable support number there is, and your helpdesk may already collect it — check before you ask twice.
- How would you rate the support you received?A scale from 1 to 5
- Anything else you want to tell us?Free text · optional
Effort, after an AI conversation
What it measures. How hard people had to work to get it resolved, one to seven.
When to ask it. After an AI answer. A wrong answer and a right one that took four attempts both score badly, which is the thing you need to know about an AI.
- How easy was it to get your issue resolved?A scale from 1 to 7
- Anything else you want to tell us?Free text · optional
Would they recommend you
What it measures. How likely people are to recommend you, zero to ten.
When to ask it. Of customers rather than of a conversation, at most twice a year. It is about the relationship, not the last ticket, and it needs dozens of answers before it means anything.
- How likely are you to recommend us?A scale from 0 to 10
- What is the main reason for your score?Free text · optional
Thumbs up or down
What it measures. The share of people who say the answer helped.
When to ask it. In a widget or in the product, where a scale is too much to ask. It tells you how many, not why, so keep the free-text question with it.
- Did that answer help?Yes, it helped · No, it did not
- What was missing?Free text · optional
Did this fix it?
What it measures. Whether the person agrees their problem is resolved — the customer’s own answer, which outranks anything we judged.
When to ask it. After a follow-up, or once a fix has gone out to the people who hit the problem. It is the customer’s own word on whether it is resolved, which outranks any judgment of yours.
- Is this resolved for you now?Yes, it is fixed · Partly · No, it is not
- What is still happening?Free text · optional
One targeted question
What it measures. Which of a few explanations is the real one, when the evidence cannot tell them apart.
When to ask it. When a problem has two or three possible causes and you are about to guess. Replace the answers with the explanations you are choosing between.
- What were you trying to do?The first thing · The second thing · Something else