Model Providers
Connect the AI model providers AskRAI runs on — every model routing can choose from is one you register here
The Model Providers section is where you connect the AI providers AskRAI runs on. Every model the platform can use is one you register here — AskRAI ships none of its own — so this page is a prerequisite for answering queries at all, not an optional extension. Once connected, these models appear in Model Routing configuration, where they can be assigned to routing rules. You can also check any connected model on its own — directly from its provider card — using Test Model.

Each connected provider is listed with its type, status, and the models you enabled on it, each with a button to test it.
Until one provider is active with at least one enabled chat model, nothing that needs a model works. A decision model answers no query on its own. Queries cannot be answered, and the Sandbox replaces its chat with a Configure a model provider empty state. Set this up first.
Model Providers opens to the holders of View model providers and Manage model providers, Platform Admins among them. View reads every provider, its models and usage read-only, never a stored key; Manage connects providers and models, reveals their stored keys and tests a model. Anyone else does not see Model Providers in the Admin sidebar, and one who follows a direct link to it is sent back to the Projects workspace. Roles are assigned under Admin → Access.
Supported provider types
| Provider Type | Credential Fields | Configuration Fields | Model kind |
|---|---|---|---|
| OpenAI | API Key | — | Chat |
| Anthropic | API Key | — | Chat |
| Azure OpenAI | API Key | Base URL, API Version | Chat |
| AWS Bedrock | An IAM role (recommended), or AWS Access Key ID and AWS Secret Access Key | Region | Chat |
| GCP Vertex AI | Service Account JSON | Project, Region | Chat |
| Custom (OpenAI-compatible) | API Key | Base URL | Chat |
| TypeSafe | API Key | Base URL (optional) | Decision |
Chat and decision models
Every model has a kind, chat or decision. It follows from the provider's type, as the table shows, and cannot be set on the model.
- A chat model writes text. It answers questions, refines them, judges guardrails, assesses confidence and runs agents.
- A decision model writes no text. It answers yes/no and score questions about a text, each with a probability. TypeSafe's Jev models are decision models.
A decision model is offered only where AskRAI makes a decision: the auto-route Classifier Model, the Confidence Assessment Model, a guardrail's Judge model and an agent's Decision model. Every other model picker offers chat models only, and AskRAI refuses a decision model in any of them: each routing stage and rule, an agent's model and a guardrail's review model.
Custom (OpenAI-compatible) providers
Use Custom (OpenAI-compatible) for any endpoint that speaks the OpenAI Chat Completions API, such as a LiteLLM proxy, a vLLM server, a hosted gateway like OpenRouter, or a gateway you run yourself. Set Base URL to the endpoint's OpenAI-compatible root, which usually ends in /v1 (for example, https://your-gateway.example.com/v1), enter the endpoint's API key, and enter each Model ID exactly as the endpoint names the model.
The Base URL must be an https address that resolves to a public IP. AskRAI refuses one on a private, loopback or reserved address, so a gateway inside your own network has to be published before AskRAI can reach it.
AskRAI does not test any particular gateway. Answering queries, evaluating guardrails and running agents rely on structured output (OpenAI's JSON-schema response_format), and agents also rely on tool calling. Whether those work through an OpenAI-compatible endpoint depends on the model and the host it routes to. Test Model sends one plain prompt, so it confirms the endpoint answers but checks neither.
AWS Bedrock models
On an AWS Bedrock provider, enter each Model ID as a foundation model ID (for example, anthropic.claude-3-5-sonnet-20241022-v2:0) or a cross-region inference profile ID (for example, us.anthropic.claude-sonnet-4-5-20250929-v1:0). OpenAI's models on Bedrock are entered the same way: a GPT model through its inference profile (for example, us.openai.gpt-6.1-sol), and the open-weight gpt-oss by its model ID (for example, openai.gpt-oss-120b-1:0). A cross-region inference profile runs its model in more than one region, so your IAM role needs access to the model in each of them: see the permissions policy.
A model named by an ARN is not supported, and AskRAI refuses one when you add it. That rules out a model that has only an ARN: an application inference profile, a provisioned throughput or an imported model.
GCP Vertex AI service account keys
In Service Account JSON, paste the JSON key file Google issued for the service account, as you downloaded it. AskRAI refuses a key that is not a service account key, and a key whose token_uri or universe_domain names a host other than Google Cloud's. A key from another universe, such as a sovereign cloud, is not supported.
TypeSafe providers
Use TypeSafe to connect TypeSafe's Jev decision models with your own TypeSafe account. Enter your TypeSafe API key. Leave Base URL blank to reach TypeSafe's own API at https://api.typesafe.ai. A Base URL you do set must be an https address that resolves to a public IP, as on a Custom provider.
Enter each Model ID as a versioned id, ending in a full version number such as jev-1.14.0. AskRAI refuses any other name, an alias such as jev-latest or a partial version such as jev-1.14 among them. An alias moves to each new version TypeSafe releases, so the same question could start getting different answers with nothing changed on your side. To move to a new version, edit the model and change its Model ID: the classifier, the confidence assessment model, and every guardrail and agent that name it move with it.
A new TypeSafe model's Input $ / 1K tokens and Output $ / 1K tokens are filled in at TypeSafe's list price: 0.000042 for input and 0 for output, which TypeSafe does not charge for. Change them if your contract with TypeSafe says otherwise. Project budgets meter agents' beats, so the decision calls a beat makes, its agent's decision model checks and its decision judges, count toward its project's budget at these prices. A person's question is metered by no budget, whichever decision model it reaches.
What AskRAI sends to a TypeSafe model is processed under your own agreement with TypeSafe. For auto-route, that is the question; for the confidence assessment, the question, its sources and the answer; for a guardrail's judge, the text it judges; and for an agent's checks, its name and role, its inbox and task titles. TypeSafe does not train on customer data, and offers zero data retention only to its enterprise customers. TypeSafe's Data Processing Agreement sets out the terms.
Text written to steer a decision model can move its answers. Whoever writes the text it reads can push it toward an answer. For auto-route, that is the person asking the question, who could word it to land in another tier. For an agent's checks, it is whoever messages the agent or names a task. Do not rely on a decision model as a defence against an attacker.
Adding a provider
Open Model Providers
Open the Admin plane from the top navigation bar, then select Model Providers in the Admin sidebar.
Click Add Provider
Click Add Provider in the top-right corner. The Add Model Provider dialog opens.

The dialog adapts its credential and configuration fields based on the selected provider type.
Enter Provider Details
Fill in the required fields:
- Provider Name — a display label for this provider, at most 120 characters (for example, "Production OpenAI" or "Internal Azure GPT-4"). A name another of your providers holds is refused under the field, on create and on a rename
- Provider Type — select from the dropdown (OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, GCP Vertex AI, Custom (OpenAI-compatible), or TypeSafe)
- Credentials — enter the required credentials for the selected provider type. Credentials are encrypted at rest and are never included in the provider list or detail responses (those return only which fields have been set, never the values). An AWS Bedrock provider first asks how AskRAI authenticates: through an IAM role, which stores no credentials and is recommended, or with access keys.
If your provider type requires additional configuration (such as a Base URL for Azure OpenAI or a Region for AWS Bedrock), those fields appear automatically.
Add Models
Click Add Model in the Models section. For each model, enter:
- Model ID — the provider's model identifier (for example,
gpt-4-turboorclaude-3-5-sonnet-20241022, or a versioned id such asjev-1.14.0on TypeSafe) - Display Name — a human-readable label shown, followed by the provider's name in parentheses, in routing configuration, every model picker and wherever the console names an agent's model. Each model on a provider needs a display name of its own, compared regardless of capitals. Two providers may each offer a model of the same name: the provider's name tells them apart
- Input $ / 1K tokens and Output $ / 1K tokens — what this model costs you, in US dollars per 1,000 tokens. Both are optional; see Token pricing
Click Add to save the model. You can add multiple models to a single provider.
A chat model is not tied to a stage of the pipeline. Every enabled chat model is offered in every stage's select on Model Routing, and you decide there which model serves each stage. A decision model is offered only as the auto-route classifier and the confidence assessment model (Chat and decision models).
Create the Provider
Click Create provider to save. The provider appears as a card in the Model Providers section with its status and model count.
Connecting AWS Bedrock with an IAM role
An AWS Bedrock provider can reach your models through an IAM role in your AWS account that AskRAI assumes, instead of through access keys. This is the recommended way to connect Bedrock: AskRAI stores no keys or secrets for the provider, so there is no long-lived credential to rotate or leak. Access keys remain supported (Connecting with access keys).
Choose IAM role in the provider form. In the Add Model Provider dialog, set Provider Type to AWS Bedrock, then choose IAM role as its authentication. The form shows two values to copy into your AWS account:
- AskRAI's assumer role ARN — the AWS role AskRAI assumes your role from.
- Your external ID — unique to your tenant. AskRAI generates it, and it cannot be chosen or changed.
Create a role with a permissions policy. In your AWS account, create an IAM role, with any name. Give it a permissions policy that allows bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream on the foundation models AskRAI will use. For a cross-region inference profile (a model ID such as us.anthropic.claude-…), allow them on the inference profile's ARN as well, and on the foundation model's ARN in each region the profile routes to:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream"
],
"Resource": [
"arn:aws:bedrock:*::foundation-model/anthropic.claude-*",
"arn:aws:bedrock:<region>:<account-id>:inference-profile/us.anthropic.claude-*"
]
}
]
}The first resource covers Anthropic's Claude foundation models in every region, so it includes each region an inference profile routes to; narrow it to the models you use. In the second, <region> is the region you set on the provider and <account-id> is your AWS account ID. For OpenAI's models, add the same two resources with openai.gpt-* and us.openai.gpt-* in place of the Claude patterns.
Model access must also be enabled for the account in the Amazon Bedrock console. The permissions policy lets the role call a model; it does not enable the model. For Anthropic's models, first submit Anthropic's use-case form. Then have an administrator open each model once in the Bedrock console's playground: that first call subscribes the account through AWS Marketplace, which the role cannot do.
Add the trust policy. Give the role a trust policy whose principal is AskRAI's assumer role. It admits the assumer only with your external ID, and lets it tag the session with your tenant ID:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "<assumer role ARN from the form>"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "<your external ID>"
}
}
},
{
"Effect": "Allow",
"Principal": {
"AWS": "<assumer role ARN from the form>"
},
"Action": "sts:TagSession",
"Condition": {
"ForAllValues:StringEquals": {
"aws:TagKeys": ["askrai-tenant-id"]
}
}
}
]
}The external ID is what keeps the role yours: it stops another AskRAI customer from using your role, even one who learns its ARN. Match it exactly, with StringEquals: a pattern such as askrai-* would admit every customer's ID, and the save refuses such a role. Tagging a session assumes nothing on its own, so the second statement needs no external ID.
Enter the role in AskRAI and save. Back in the form, paste your role's ARN, set Region, add your models and click Create provider. When you save, AskRAI checks the trust: it assumes your role with your external ID, and confirms that the role refuses a different external ID. If either check fails, the save is refused with the reason, for example a trust policy without the sts:ExternalId condition.
Test a model. The save checks the trust, not model access. Test a model on the provider to confirm AskRAI can call it: the test makes one small Bedrock call.
Sessions in your AWS account
Every session AskRAI opens with your role is named askrai-<your tenant ID> and carries the session tag askrai-tenant-id, so AskRAI's calls are visible in CloudTrail under that name. You may use the session name and the tag in conditions of your own policies.
Connecting with access keys
To connect with access keys instead, choose them as the provider's authentication in place of IAM role, and enter an AWS Access Key ID and AWS Secret Access Key allowed the same Bedrock actions as the permissions policy above. They are encrypted at rest, as every provider credential is.
Provider card
Each connected provider displays as a card showing:
| Element | Description |
|---|---|
| Provider icon and name | Visual identifier with the provider type label below |
| Status chip | Active (green), Inactive (grey), or Error (red) |
| Model count | Number of enabled models on this provider |
| Edit button | Opens the edit dialog to update settings or credentials |
| Delete button | Removes the provider and its stored credentials. Refused while an agent, a routing entry or a guardrail runs on one of its models |
| Model list | Each enabled model with its display name and a play button to test it |
Editing a provider
Click the edit icon on a provider card to open the edit dialog. You can update:
- Provider Name — change the display label
- Status — toggle between Active and Inactive using the switch. Switching a provider inactive stops every agent, routing entry and guardrail that runs on its models until it is active again, so the dialog lists them under the switch, and Save changes waits until it has. It stays allowed, as an emergency stop after a leaked key, for example
- Credentials — leave blank to keep current values, or enter new credentials to replace them. Click the eye icon to reveal a stored credential value on demand.
- Configuration — update Base URL, API Version, Region, or Project as applicable
- Models — add new models, edit or remove existing ones, and put them in order
Editing a model
Click the pencil icon beside a model in the edit dialog to change its Model ID, Display Name, Input $ / 1K tokens, Output $ / 1K tokens and whether it is Enabled. Click Done, then Save changes. The model keeps its place on the provider, and a change of its display name, prices or Enabled takes effect on its next use.
Correcting a model ID — for a typo, say, or a provider renaming a model — changes the ID every agent, routing entry and LLM-judge guardrail on the model refers to. When anything runs on it, the row lists what does once you click Done, and Save changes moves them all to the corrected ID. An agent turn or a query already under way as you save can fail once on the old ID; the next one runs on the corrected model.
Ordering a provider's models
A provider's models are listed in the order you give them, in its edit dialog and on its card, and every model picker in the console lists them in that order too: an agent's model and decision model, a new team member's, an agent made from a template, a template's model, a guardrail's judge and review models, and every stage of Model Routing. A disabled model keeps its place on the provider but is not offered in any picker. A model you add goes last, and the models of a new provider can be put in order the same way in the Add Model Provider dialog before you create it.
To move a model, drag it to its new place in the edit dialog, or press its Move up or Move down button, then click Save changes. The new order is saved with the provider's other changes and recorded in the Change Log. If another admin added or removed one of the provider's models while your dialog was open, the order is not saved: the dialog says so and opens again on the provider as it now is, so you can put its models in order again.
Each provider's models stay together in a picker. The order of the providers themselves is not set here.
Removing or disabling a model in use
A model that an agent, a routing entry (in the published routing configuration or its draft) or a guardrail runs on cannot be removed or disabled: each of them would fail on its next use. Clicking Remove, or switching Enabled off and clicking Done, shows what runs on the model instead. Point each of them at another model, or delete it, and then remove or disable the model. A provider whose models are in use cannot be deleted either, and its delete confirmation lists what uses them.
The provider type cannot be changed after creation. To switch provider types, delete the existing provider and create a new one.
Testing a model
Click the play icon next to any model in a provider card to open the test drawer. Enter a prompt and click Send Test to verify the model responds correctly. The result shows:
- Success — the model's response text and latency in milliseconds
- Failed — the error message from the provider
A decision model writes no response text, so its test drawer asks for a yes/no question and a text to ask it about. A success shows the probability that the answer is yes, the versioned model that answered, and the latency.
Credentials are encrypted at rest and are never included in the provider list or detail responses — those carry only which fields have been set, never the values. When editing a provider, existing credentials appear as masked dots; a Model Providers administrator can reveal a stored value on demand by clicking the eye icon, which fetches it through a dedicated, permission-gated endpoint.
Token pricing
When you bring your own model, AskRAI does not know what your contract with the provider costs. Input $ / 1K tokens and Output $ / 1K tokens are where you tell it — and they are what makes project budgets accurate.

Token prices sit alongside the model's identifier in the Add Model form. Both are optional — leave them blank and the model bills at the platform's fallback rate.
Both fields take US dollars per 1,000 tokens — so a model billed at $2.00 per million input tokens is entered as 0.002. They are available on every provider type.
| Field | Description | Required |
|---|---|---|
| Input $ / 1K tokens | What you pay for 1,000 prompt tokens | No — leave blank and beats are costed at the flat fallback rate |
| Output $ / 1K tokens | What you pay for 1,000 completion tokens | No — leave blank and beats are costed at the flat fallback rate |
- Leave a price blank and that side of the model bills at the platform's fallback rate. The two sides fall back independently, so you can price only the one that matters.
- Zero is a valid price. Enter
0for a model that genuinely costs you nothing — a self-hosted deployment, for example. A negative number is rejected with Token prices must be zero or a positive number. - A TypeSafe model starts at TypeSafe's list price, which you can change. See TypeSafe providers.
- Once a model is saved, its row summarizes the rates as · in $0.002 / out $0.008 per 1K, and shows platform rate for whichever side you left blank.
Prices are used for budget accounting, and only for that — they do not influence routing or which model handles a query. Getting them wrong does not break anything, but it does make every project budget wrong in the same direction: a model priced too low will overspend its allocation before the budget stops it.
Pricing is read fresh on every agent beat, so correcting a rate takes effect immediately — there is no cache to wait out and no need to restart a running team. To correct a saved model's price, edit the model.
Related pages
- Model Routing — configure which models handle different query types using your connected providers, and which model classifies each question for auto-route
- Confidence Thresholds — set the score boundaries that band an answer and drive escalation. They do not feed routing, whose conditions are the request-context fields listed on that page
- Sandbox — test AI responses with different model configurations