Confidence & Escalation
How AskRAI scores response confidence and uses escalation rules to decide when conversations need human attention
Every AI response receives a confidence score — a percentage reflecting how well the knowledge base supports the answer. The score sorts the answer into a band, and your escalation rules decide what follows from that band. Scoring happens as the answer is produced; escalation happens after it has been delivered, which is what decides the difference between the two.
Confidence bands
Confidence scores fall into three configurable bands. You set the thresholds in Settings — the defaults are shown here.
| Band | Default range | Typical behavior |
|---|---|---|
| High | 90% and above | The assistant is confident in its answer. Response is delivered normally. |
| Medium | 60% – 89% | The assistant has a reasonable answer but some uncertainty. The response is delivered. |
| Low | Below 60% | The assistant cannot adequately answer. The conversation is typically escalated. |
Threshold boundaries are fully configurable per tenant. When you adjust them, a reclassification preview shows how existing conversations would be recategorized — so you can see the impact before saving.
Escalation Rules
Escalation rules define what happens when a response falls into a particular confidence band. Each rule specifies a condition (confidence band) and an action to take.
Four escalation actions
| Action | What it does |
|---|---|
| Create Ticket | Generates a support ticket in the Tickets queue for human follow-up. The ticket carries the original question, and — unless you clear Include AI context on the rule — the AI response and its confidence score. |
| Flag for Review | Not yet implemented. Selectable today, but no conversation is marked and no one is notified. |
| Suppress Response | Not yet implemented — and by design it could not block a response anyway, because escalation is evaluated only after the answer has been delivered (see below). |
| Auto Respond | Delivers the AI response without any escalation, regardless of confidence. Use this for bands where you trust the assistant's judgment. |
Escalation happens after delivery, and Create Ticket is the only action that currently does anything. AskRAI answers the user first, and your rules are then evaluated against the record of that turn. So escalation is a follow-up mechanism, not a gate — nothing it does can un-send an answer. To prevent a response, use a guardrail, which is evaluated during the turn itself.
One other thing can stop an answer before it is written, and it is neither a guardrail nor an escalation rule: if the best knowledge-base match scores below the tenant's minimum relevance threshold, the assistant returns its low-confidence reply without composing an answer at all. See Confidence Thresholds.
Rule ordering
When multiple escalation rules match (for example, if two rules both apply to the Medium band), the rules are evaluated in priority order. The first matching rule's action is executed.
Every confidence band should have at least one matching escalation rule. If no rule matches a given band, the platform defaults to delivering the response without escalation — which may not be the behavior you want for low-confidence answers.
Confidence in practice
The confidence score is not a simple keyword-match percentage. After the assistant drafts an answer, a lightweight assessment step scores two things:
- Answer quality and groundedness — how well the drafted answer actually addresses the user's question using the retrieved knowledge base content, rather than guessing or padding.
- Retrieval relevance — how close the top knowledge base results the answer was built from are to the question, measured as the similarity score retrieval returns for each of them.
Only the first of those bands the answer. Retrieval relevance is kept as a separate signal and does not move the confidence band. A high score means the assistant judged its own answer well-grounded in what it retrieved. A low score means it is uncertain — perhaps the question is outside the knowledge base's scope, or the best-matching content only partially fits.
Testing Escalation Rules
Use the Sandbox to test how confidence scoring and escalation rules behave with real queries. The sandbox's audit preview tab shows:
- The confidence score the query received
- Which escalation rule matched
- What action would be taken in production
This lets you tune thresholds and rules before they affect live conversations.
Next steps
- Settings — configure confidence thresholds and escalation rules
- Tickets — view and manage escalated conversations
- Sandbox — test confidence and escalation behavior
- Governance & Audit — learn how confidence data feeds into analytics