Resources

Templates

The questions we ask customers, and a scorecard for checking your AI’s answers by hand, written out to copy. Each says what it measures and when it is worth asking, which matters more than the wording.

AI answer QA scorecard

Questions to ask of one answer your AI gave, each answered yes, no, not applicable. Before you use one, write beside it what it rests on: the passage of your documentation, or the rule your team states. A criterion that rests on nothing turns a review into opinion, and two reviewers will not agree on it.

GroupCriterionRests on
RightThe answer is correct according to your documentation as it stood when the answer was given.The article or passage it should have matched
RightEvery policy, price or commitment it states is written in your documentation.The passages that state them
RightIt answers the question the customer asked, not a nearby one.A rule your team states
BackedWhere it relied on an article, the customer can see which one.A rule your team states
BackedIt promises nothing your team has not agreed to do: no refund, callback or date nobody approved.Your refund and escalation policy
HandledIt is written in the language the customer wrote in.A rule your team states
HandledWhere it could not answer from your documentation, it handed over to a person instead of guessing.Your handover rule
HandledAfter a handover, the customer did not have to explain the problem again.A rule your team states

Download the spreadsheet (CSV)

Count the people behind a result, not the conversations, and say how many answers you looked at beside any share you report: a few reviews are an anecdote, however clean the percentage looks.

Survey templates

One or two questions each. A short survey is the one people answer; where there is a second question, it is optional free text, because a number says how many and only words say why.

Satisfaction, after a conversation closes

What it measures. How people rate the help they got, one to five.

When to ask it. After a conversation closes, about that conversation. It is the most comparable support number there is, and your helpdesk may already collect it — check before you ask twice.

  1. How would you rate the support you received?A scale from 1 to 5
  2. Anything else you want to tell us?Free text · optional

Two questions, under a minute.

Effort, after an AI conversation

What it measures. How hard people had to work to get it resolved, one to seven.

When to ask it. After an AI answer. A wrong answer and a right one that took four attempts both score badly, which is the thing you need to know about an AI.

  1. How easy was it to get your issue resolved?A scale from 1 to 7
  2. Anything else you want to tell us?Free text · optional

Two questions, under a minute.

Would they recommend you

What it measures. How likely people are to recommend you, zero to ten.

When to ask it. Of customers rather than of a conversation, at most twice a year. It is about the relationship, not the last ticket, and it needs dozens of answers before it means anything.

  1. How likely are you to recommend us?A scale from 0 to 10
  2. What is the main reason for your score?Free text · optional

Two questions, under a minute.

Thumbs up or down

What it measures. The share of people who say the answer helped.

When to ask it. In a widget or in the product, where a scale is too much to ask. It tells you how many, not why, so keep the free-text question with it.

  1. Did that answer help?Yes, it helped · No, it did not
  2. What was missing?Free text · optional

Two questions, seconds.

Did this fix it?

What it measures. Whether the person agrees their problem is resolved — the customer’s own answer, which outranks anything we judged.

When to ask it. After a follow-up, or once a fix has gone out to the people who hit the problem. It is the customer’s own word on whether it is resolved, which outranks any judgment of yours.

  1. Is this resolved for you now?Yes, it is fixed · Partly · No, it is not
  2. What is still happening?Free text · optional

Two questions, seconds.

One targeted question

What it measures. Which of a few explanations is the real one, when the evidence cannot tell them apart.

When to ask it. When a problem has two or three possible causes and you are about to guess. Replace the answers with the explanations you are choosing between.

  1. What were you trying to do?The first thing · The second thing · Something else

One question, seconds.