Governance & Audit
Audit trails, analytics dashboards, sandbox testing, and the continuous improvement loop for responsible AI
AskRAI is designed for organizations that need accountability, transparency, and continuous improvement in their AI operations. Every interaction is recorded, every guardrail evaluation is logged, and every confidence score is tracked — giving your team the data needed to monitor, measure, and improve your AI assistant over time.
Two different things are called governance. This page is about the record — what AskRAI captures about an interaction, and the review loop that record supports. The controls that decide what an interaction is allowed to do — policies, guardrail definitions, egress allowlists — live under Settings → Governance. The two meet here: every control that runs writes to the trail below.
The continuous improvement loop
Governance in AskRAI is not a one-time setup — it is an ongoing cycle of interaction, review, tuning, and testing.
- Interact — users ask questions through connected channels. Every interaction flows through the query pipeline.
- Audit — the platform records the full context of every interaction: the query, the response, the confidence score, guardrail evaluation results, and processing metadata.
- Review — administrators use the dashboard and conversation logs to identify patterns: frequent low-confidence answers, guardrail false positives, knowledge gaps, and usage trends.
- Tune — based on review findings, administrators adjust configuration: add knowledge content, refine guardrail prompts, adjust confidence thresholds, or modify escalation rules.
- Test — changes are validated in the sandbox environment before going live, ensuring they work as expected.
- Deploy — verified changes are applied to production, and the cycle continues.
What the audit trail captures
Every conversation generates an audit record containing:
| Data point | Description |
|---|---|
| User query | The original question the user asked |
| AI response | The generated answer |
| Confidence score | The score the answer received. The band is not stored — it is derived from your thresholds as you read the record, so moving a threshold relabels past conversations |
| Guardrail evaluations | Pass/fail result for each guardrail, with the reason any that fired gives. A check that passed records no reason |
| Ticket decision | Whether a ticket should be opened, the reason, and the resulting ticket ID |
| Processing metadata | Response time, model and routing information, channel, caller, and session details |
This data powers the dashboard analytics, conversation review, and compliance reporting.
One thing the audit record deliberately does not keep: the retrieved search results — which knowledge items matched, and their scores. Retrieval detail is available live in the Sandbox, which is built for exactly this kind of inspection but never persists anything. The escalation rule is recorded: when a rule opens a ticket, the audit's ticket-decision reason names it ("Escalated by rule: …"), though not its id.
Dashboard analytics
The Dashboard provides at-a-glance metrics for monitoring platform health:
- Request volume — total conversations and trends over time
- Response quality — success rate based on confidence thresholds
- Response speed — average processing time and distribution
- User engagement — active user counts and channel breakdown
- Confidence trends — distribution of responses across high, medium, and low bands
- Channel analytics — per-channel usage and performance
All metrics are filterable by date range and channel.
Conversation review
The Conversation Logs let administrators drill into individual interactions to:
- Review the full conversation thread with user and assistant messages
- Inspect the audit details for any message — confidence score, guardrail results, and model and routing information
- Mark guardrail triggers as false positives to track and reduce over-triggering
- Identify knowledge gaps where the assistant could not provide a good answer
Search debug data is not part of the audit record; to see which knowledge items a question retrieves, replay it in the Sandbox.
Escalation tracking
When conversations are escalated, they create tickets in the Tickets queue. Each ticket captures:
- The original question, and the AI response unless the rule clears Include AI context
- The confidence score and which escalation rule triggered
- Severity classification for triage
- Status tracking through the resolution workflow (Open → In Progress → Resolved → Closed)
Ticket patterns reveal systemic issues — if many tickets come from the same topic, it signals a knowledge gap that can be addressed by adding content.
Sandbox testing
The Sandbox is a safe environment for validating configuration changes before they affect live users. You can:
- Test with custom configurations — select specific guardrails and knowledge packs to evaluate
- Test as a group — simulate a group's exact configuration to verify the end-user experience
- Inspect the full pipeline — view guardrail evaluation results, search debug data, audit previews, and model information for every test message
- Validate escalation rules — see which rules would trigger and what actions would be taken, without creating real tickets
Sandbox conversations are not recorded as production audit records. This keeps your analytics clean while giving you full visibility into how changes will behave.
Compliance posture
AskRAI's governance model supports compliance requirements by providing:
- Traceability — every response is recorded with the question that produced it, the confidence score it received, and the result of every check that ran. Which knowledge items were retrieved is not part of that record — replay the question in the Sandbox to see the retrieval set
- Accountability — audit records capture who asked, what was answered, and what safety checks were applied
- Transparency — a guardrail that fires records why. A check that passed records no reason, and the deterministic kinds never produce prose at all — a fired pattern check records how many spans matched
- Control — the content lifecycle (Draft → Approved → Needs Attention → Discarded) decides what is retrievable, and gives you a place to park an item you no longer trust
Next steps
- Dashboard — monitor platform metrics and trends
- Conversation Logs — review individual interactions
- Tickets — manage escalated conversations
- Sandbox — test configuration changes safely
- Settings — configure thresholds and escalation rules