AutoQA: Automated Conversation Quality Assurance

Automatically score support conversations against quality rules you write in plain English

AutoQA scores every support conversation (tickets, chats, and Voice AI calls) against rules you write in plain English. Each conversation gets a QA score from 0 to 5 and a pass or fail for every rule, with explanations.

To test the LLM calls your own apps make, see AutoEval.

​

Create a rule

1. Open QA Rules and click Create New Rule.

2. Fill in the form:

   Field	What to enter
   Short Name	Optional label for tables and filters, such as “Identity verification”.
   Rule Description	What to check, in plain English. Example: “Flag conversations where the agent did not verify the customer’s identity before making account changes.”
   Category	Compliance, Tone & Empathy, Resolution Quality, Process Adherence, Escalation, or Custom.
   Severity	Critical, Warning, or Info.
   Channels	Which conversations the rule scores: Tickets, Chats, or Voice AI calls. For Voice AI calls, you can narrow to specific call types or campaigns.
   Score weight (%)	Optional. Weights are normalized per review, so they don’t need to total 100.

3. Click Create Rule. New rules are active and apply to new conversations.

​

Example rules

Category	Rule
Compliance	Agent must verify customer identity before making any account changes
Tone & Empathy	Agent must maintain a professional and empathetic tone throughout the conversation
Process Adherence	Agent should offer a follow-up or ask if the customer needs anything else before closing
Resolution Quality	AI agent must not fabricate or hallucinate product features, pricing, or policies

​

Manage rules

The QA Rules page shows Active Rules, Rule Evaluations, Avg Rule Pass Rate, and Critical Rules Below 90%. Filter the table by All, Active, or Inactive, or search.

From a rule’s actions menu (three dots):

• Edit: change the rule and see its Pass Rate (All time), Cases Evaluated, Status, and Created by. Click Save Changes.
• Disable or Enable: inactive rules stop scoring new conversations but are kept.
• Delete: permanently removes the rule and its evaluation history.

​

Review results

Open QA Reviews. The top cards show Reviewed, Failures, Critical Failures, and Avg QA Score.

Filter by channel, Result (Pass or Fail), Rule, Severity, Agent, or Agent type, or search.

Click a review to see:

• QA Scorecard: every rule checked, pass or fail, with an explanation for each failure.
• Flagged Messages: the messages that caused a violation. Chats and calls also show the full transcript.

​

Score colors

Metric	Green	Orange	Red
Pass Rate	90% or above	75% to 89%	Below 75%
QA Score	4.0 or above	3.0 to 3.9	Below 3.0

Need help writing rules? Email us.