Model Routing
Configure which AI models handle different pipeline stages and define rules to route requests based on context
Model Routing controls which AI models process each stage of the response pipeline. Configure default models, enable automatic complexity-based routing, and create rules that match requests by channel, authentication status, or user group. Changes follow a draft-and-publish workflow with full version history and rollback support.
To open it, switch to the Admin plane from the top navigation bar and select Routing in the Admin sidebar.
Routing opens to the holders of View routing or Manage routing, Platform Admins among them, and to View alone read-only. Anyone else who opens it is redirected away — see Access.

Everything below the publish bar is a draft. Nothing on this page affects a live query until you publish it.
Key concepts
Before configuring routing, understand the three layers that determine which model handles a request. They are evaluated in order of precedence:
| Layer | Description | Precedence |
|---|---|---|
| Routing Rules | Match requests by context (channel, auth status, group, role, complexity) and assign specific models | Highest |
| Auto-Route | Classify query complexity as simple, moderate, or complex, then map each tier to a model | Medium |
| Defaults | Tenant-level fallback models for each pipeline stage | Lowest |
There is no fourth layer beneath these. Every model AskRAI runs is one your tenant registered under Model Providers, so a stage that no layer assigns has nothing to fall back on — and what happens then depends on the stage. Summarization has nowhere to go without one, so the request fails; guardrail evaluation fails the same way, but only for a tenant whose guardrails include an LLM judge — the deterministic kinds need no model. Query refinement is the only stage that degrades quietly — it answers on the unrefined question. Confidence assessment fails closed: every answer it scores comes back 0.0, so the answer bands Low for the user, 0.0 is what the audit record keeps, and any Escalation Rule on the Low band fires on that conversation. It scores every generated answer — a curated answer served straight from the knowledge base is banded from its own score and is unaffected. The retrieval relevance score is recorded alongside it but is not what bands the answer. Set a tenant default for every stage you rely on.
Pipeline stages
Each request passes through up to four pipeline stages. You can assign a different model to each:
| Stage | Purpose | Typical Model |
|---|---|---|
| Summarization | Synthesizes the final response from search results | High-quality model (e.g., GPT 5.2) |
| Query Refinement | Resolves conversational references into self-contained queries | Fast model (e.g., GPT 4.1 Mini) |
| Guardrail | The model an LLM-judge guardrail puts its rule to, at whichever stage that guardrail reads | Fast model (e.g., GPT 4.1 Nano) |
| Confidence Assessment | Scores how well the retrieved knowledge supports the answer | Fast model (e.g., GPT 4.1 Nano), or a decision model |
Draft and publish workflow
Routing configuration uses a draft-and-publish workflow. Changes you make on the page are local edits until you save them as a draft. The draft must then be published to take effect in production.
The publish bar at the top of the page shows:
| Control | Description |
|---|---|
| Validate | Checks the configuration for errors — for example a rule that sets no model at all. A rule with no target model is valid provided it overrides at least one stage. A configuration that leaves some queries without a Summarization Model is valid, with a warning that names the stage and sends you to Default Models; you can still publish it. A configuration with neither says Configuration is valid |
| Save draft | Persists your edits as a draft without affecting production |
| Discard | Reverts all unsaved local edits back to the last saved draft |
| History | Opens the version history drawer to compare or rollback versions |
| Publish | Promotes the saved draft to production (disabled until a valid draft is saved) |
Another admin may save or publish the routing config while you have the page open. Save draft and Publish then refuse your change rather than write over theirs, and the page says so with a Reload button. Reload to see their version, then make your change again: reloading discards your unsaved edits.
Publishing a routing configuration takes effect immediately for all new requests in your tenant. Test changes in the Sandbox before publishing to production.
Default models
The Default Models section sets tenant-level fallback models for each pipeline stage. If no routing rule or auto-route tier assigns a model for a stage, the default is used.
Select a model from the Summarization Model dropdown to set the primary default. Expand the Advanced section to override the Query Refinement, Guardrail and Confidence Assessment models separately.
The Assistant Model is the chat model a project's assistant gets when it is added. Left on None, an assistant gets the Summarization Model instead. An assistant keeps the model it got, so changing this later moves only the assistants added after; one added while neither model was set answers on whichever is set when it's asked, and can't be asked while both are on None.
Leave a dropdown on None and that stage has no tenant default at all. No pipeline stage falls through to the Summarization Model — it is the summarization stage's own default and no other stage's; only assistants, which are no stage of a query, take it. The single exception among the stages is Confidence Assessment, which reuses the Query Refinement Model when its own is left on None. Routing rules and auto-route tiers fall through to these defaults, which is why their dropdowns offer No override; nothing falls through beneath the defaults, which is the state the note above describes.
Summarization is the stage a question cannot pass without a model, so its empty choice reads None — queries that reach this stage fail. Routing and the Admin overview both warn that some queries cannot be answered while nothing is published, or while the published routing sets no Summarization Model and auto-route is off, has no Classifier Model, or leaves a tier without a model. The Guardrail Model's empty choice names the one question it fails: one governed by an LLM judge with no judge model of its own.
Every stage's model dropdown lists every enabled chat model from your tenant's Model Providers, tagged with the provider each came from. A chat model is not tied to a stage, so any of them can serve any stage. The Classifier Model and Confidence Assessment Model dropdowns list decision models as well (A decision model as the classifier, as the confidence assessment model). A dropdown is disabled with nothing in it only while your tenant has no enabled model it can offer — register a provider, or enable a model on one you already have.
A decision model as the confidence assessment model
The Confidence Assessment Model can be a decision model connected under Model Providers, such as TypeSafe's Jev. It then scores every generated answer, streamed or not, as it is delivered: after your Output guardrails have run on it, so an answer one of them blocks is never sent to it. It reads the question, the knowledge the answer was written from and the answer, and gives the probability of two things:
- The answer addresses the question.
- The sources state every factual claim in the answer.
The answer's confidence score is the product of the two, banded against your confidence thresholds.
- A streamed answer is scored by it in place of a chat model's assessment. If it fails, the answer scores
0.0and bands Low, as it does when a chat model's assessment fails. - An answer that is not streamed takes its score in place of the one the Summarization Model gives its own answer. If it fails, the answer keeps the Summarization Model's score.
- The audit record keeps the score, the thresholds it was measured against and the versioned model that answered, such as
jev-1.14.0.
A curated answer served straight from the knowledge base is banded from its own score, as before. Each question, its sources and its answer are processed under your own agreement with TypeSafe.
Choosing a decision model moves which band your answers fall in, and so the escalation tickets your rules open. Its score measures something other than a chat model's, so the same answer can land in another band. Review your confidence thresholds and Escalation Rules when you switch, and watch the dashboard for a day or two.
Auto-route
Auto-route automatically classifies each incoming query by complexity and routes it to the most appropriate model. This optimizes cost and performance by sending simple queries to faster models and reserving powerful models for complex requests.

The classifier is itself a model call on every query, so it is worth pointing at a cheap, fast model — its only job is to decide which tier the question belongs to.
Enabling auto-route
Toggle Enable auto-route classification to activate this feature. When enabled, two additional configuration areas appear:
- Classifier Model — the model that evaluates query complexity: a chat model, or a decision model such as TypeSafe's Jev. Every query pays its latency, so a small, quick model is the right choice.
- Tier Mapping — assign a summarization model (and optionally query refinement and guardrail models) for each complexity tier.
Tier configuration
| Tier | When Used |
|---|---|
| Simple | Straightforward factual queries that require minimal reasoning |
| Moderate | Multi-step questions that need some analysis or comparison |
| Complex | Queries requiring deep reasoning, multi-document synthesis, or specialized knowledge |
For each tier, select a primary model from the dropdown. Expand Advanced under each tier to set stage-specific overrides for query refinement and guardrail models.
Whenever the classifier fails — a timeout, a refused key, an error from its provider — the question is routed as Moderate rather than failed.
A decision model as the classifier
The Classifier Model can be a decision model connected under Model Providers, such as TypeSafe's Jev. It writes no text. It scores the question as simple, moderate or complex, with a probability for each and a confidence in the result.
- The tier is the one with the highest probability.
- Below a confidence of 0.5, the tier is Moderate, as it is on any failure. Routing is there to save cost, and a question the classifier is unsure about goes to the middle tier.
- The audit record keeps the tier, its confidence and the versioned model that answered, such as
jev-1.14.0.
When the classifier is a TypeSafe model, each question it reads is processed under your own agreement with TypeSafe. A question written to steer the classifier can move its tier, so a decision model is no defence against an attacker.
The classifier reads the question before any guardrail. Its tier picks the guardrail model, so it has to run first. An Input guardrail that Redacts therefore never covers what the classifier reads: the question reaches the classifier's provider as it was asked. That holds for a chat classifier as well as a decision one.
Routing rules
Routing rules let you assign models based on request context. Rules are evaluated by priority — lower numbers run first. Multiple matching rules can contribute different stages (one rule might set the summarization model while another sets the guardrail model).
Creating a rule
Open the Rule Editor
Click Add Rule in the top-right corner of the Routing Rules section.

The rule editor, filled in — a rule that routes Teams requests to one model. The Target Model list offers only the models you registered under Model Providers.
Set the Rule Name and Priority
Enter a descriptive name for the rule. Set the Priority number — rules with lower priority values are evaluated first. Toggle Enabled to activate or deactivate the rule without deleting it.
Choose the Target Model
Select a Target Model (Summarization) from the dropdown. This is the primary model assignment for matching requests. The dropdown offers No override plus every enabled chat model your tenant registered — there is no platform-supplied entry to pick, and the control is disabled outright while your tenant has no enabled chat model.
Expand Stage Overrides to optionally set different models for query refinement and guardrail stages.
Define Conditions
Add one or more conditions that a request must match for this rule to apply. All conditions must match (AND logic).
Give every condition a value. While one has none, Add rule (or Update rule) stays disabled and the editor's footer reads Give every condition a value to save the rule. A draft that already holds such a condition is refused by Save draft and Validate, with Condition must have a value on that condition.
Save the Rule
Click Add rule to add the rule to your configuration. The rule appears in the list sorted by priority.
Condition fields
Each condition matches a request attribute against a value:
Each field below (except Group ID) is a fixed dropdown — these are the complete option lists, not examples:
| Field | Description | Values |
|---|---|---|
| Channel | The communication channel the request arrived on | Teams, Web Chat, WhatsApp, SMS, Phone, Email, Mobile App, API, CLI, MCP |
| Auth Status | How the caller authenticated | Authenticated, Unauthenticated, API Key, Executive SSO, Service Credential |
| Group ID | The groups the caller resolved into | Free-text group ID |
| Role Category | The categories of the roles the request matched | Public Access, Employee Access, Admin Access, API Access, System Access |
| Complexity Tier | The auto-route complexity classification (requires auto-route enabled) | Simple, Moderate, Complex |
Group ID matches against every group the caller resolved into — those assigned to them directly, and those reached through a role whose mapping rules matched the request. Automatic groups are not among them: nothing supplies the user attributes those rules read, so an automatic group never contributes to routing. Role Category matches the categories of the roles the request matched, whether or not those roles lead to a group. A caller who resolves into no groups at all is a real answer, not a blank: a not_in rule is satisfied by it.
Withdrawn conditions
Device Type is no longer a condition field. No caller ever reported a device type, so a condition on it matched no request, and a rule holding one never routed anything.
A rule saved while the field was offered still holds its Device Type condition, and it still matches no request. Removing the condition silently would let the rule start matching on its other conditions and route traffic to a model nobody chose for it, so the condition stays until you remove it:
- In the rules list, the rule is marked Withdrawn conditions: matches no request.
- The rule editor lists them under Withdrawn conditions, as they were saved. They cannot be edited.
- Saving other changes to the rule keeps them, and so do publishing and rolling back to a version that holds them.
To make the rule match again, open it, tick Remove these conditions when saving, click Update rule, then Save draft and Publish. From then on the rule matches on its remaining conditions, so check that those describe the requests you mean it to route. A rule that holds no other condition needs one before it can be saved: a rule with no condition would match every request.
Operators
| Operator | Description |
|---|---|
| Equals | Exact match against a single value |
| Not Equals | Matches any value except the specified one |
| In List | Matches any of the specified values |
| Not In List | Matches none of the specified values |
Rules override auto-route and defaults. If a rule assigns a summarization model, it takes precedence over the auto-route tier mapping and the default model for that stage.
Testing rules
The Test Rules section at the bottom of the page lets you simulate a request and see which rules match and which models are resolved for each stage.
Select values for the context fields (channel, auth status, group ID, role category, and complexity tier if auto-route is enabled), then click Evaluate. The result shows:
- Source — where the final model selection came from (Tenant Default, Routing Rule, Auto-Route, Merged Rules, or Not configured when nothing assigned the stage a model)
- Resolved Models — which model handles each pipeline stage (summarization, refinement, guardrail) and the source of each assignment
- Matched Rules — which rules matched the test inputs, sorted by priority
- Rules With Withdrawn Conditions — the enabled rules that match no request because they hold a withdrawn condition, whatever the inputs
The rule tester evaluates against your current unsaved configuration, so you can try changes before saving or publishing. It is a separate implementation from the runtime, and one difference is worth knowing. Leaving a field blank means "not supplied" in both, so a blank never matches — but the tester has no way to say "resolved into no groups", which in production does satisfy a not_in rule.
Version history
Click History in the publish bar to open the version history drawer. Each published configuration is stored as a numbered version. Publishing a draft that matches the published configuration adds no version: the draft is cleared, and the published version keeps its number.
For each version, you can:
- Compare an archived version with the current published version to see what changed
- Rollback to restore an archived version as the active configuration
Rolling back to an earlier version permanently deletes every routing version newer than the rollback target. This action cannot be undone. Other versioned areas behave differently — rolling a skill back keeps its history and republishes the older content as a new version.
Related pages
- Admin — configure confidence thresholds and escalation rules
- Sandbox — test AI responses with different model configurations
- Conversation Logs — review audit records showing which models and rules were used
- Guardrails — configure safety policies evaluated by the guardrail model
- Users, Roles & Groups — manage groups and roles used as routing rule conditions