Auditing Chat AI Employees
The Audit tab in chat AI Employees provides comprehensive capabilities to track, evaluate, and continuously improve your AI Employee's performance with access to real user conversations. This enables reviewers and builders to monitor conversations, add notes, and measure performance objectively to update the GWE workflow and AI Employee configuration.
User Roles
The Audit tab is available to users with the AI Employee Admin, AI Employee Manager, or AI Employee Metrics Viewer role on the AI Employee.
All three roles can:
- View all conversations for the AI Employee, filtered by date range.
- Filter conversations by: user dissatisfaction, user feedback added, or reviewer comments added.
- Search conversations by email, conversation ID, or message snippet.
- Open a conversation to view the full message history, user context, and conversation sources.
- Set or update a conversation's resolution status (New, Opened, Needs Improvement, Needs Clarification, Resolved).
Benefits
- Complete Conversation Visibility -- Access and review all user interactions with detailed conversation history, user context, and source attribution.
- Structured Feedback -- Capture both end-user feedback and expert notes through filters.
- Performance Optimization -- Identify patterns in successful and dissatisfying interactions to guide improvements.
Key Features
Conversation Table
The Audit dashboard provides a conversation table with:
- Conversation ID (with quick copy)
- User email
- Date
- Resolution status
- User feedback rating
Conversation Status Filtering
View and filter conversations by key criteria:
- User dissatisfaction -- Includes workflow failures, external action failures, and user negative feedback.
- User feedback added -- All conversations with any user feedback.
- Reviewer comment added -- All conversations with any reviewer comments.
Additional Filtering and Search
- Filter conversations by date range.
- Search by user email, conversation ID, or content snippets.
Detailed Conversation Review
Access complete conversation history with user context, sources used, and all feedback along with timestamps.
Adding Comments
Reviewers can provide optional comments for each AI Employee response.
Status Tracking
Mark conversations as:
- New
- Opened
- Needs Clarification
- Needs Improvement
- Resolved
Feedback Categories
Conversations are automatically categorized based on the percentage of queries:
| Category | Criteria |
|---|---|
| Good | >80% positive or no-feedback queries |
| Neutral | 50--80% positive or no-feedback queries |
| Bad | <50% positive or no-feedback queries |
Advanced Debugging
Access Show Work functionality to view intermediate agent processing and decision-making. For details, see Debug Logs.
How to Use
Getting Started
- Automatically enabled for all deployed chat-based AI Employees.
- No additional configuration required -- conversation logging begins immediately upon the first query.
- Access through the dedicated Audit tab in your chat-based AI Employee.
Pre-Launch Testing
Reviewers can simulate various user scenarios and provide structured feedback to builders during the UAT phase.
Post-Launch Operations
- Daily Monitoring -- Reviewers check recent conversations for issues requiring attention.
- Regular Evaluations -- Conduct unbiased performance assessments using randomized conversation samples.
- Continuous Improvement -- Builders monitor comments, implement fixes, and track resolution progress.