Evaluating AI Employees
Evaluations allow you to measure how an AI Employee performs against a large dataset using defined scoring criteria. This helps validate response quality, consistency, and alignment with expected behavior before or after launch.
Supported AI Employees
Evaluations are currently supported only on chat-triggered AI Employees.
Key Concepts
| Concept | Description |
|---|---|
| Evaluation | A single execution that runs against a dataset and scores each response using defined criteria. |
| Dataset | A CSV file containing test queries. Each row represents a single workflow run. |
| User Query Column | The dataset column whose values are used to trigger AI Employee workflow runs. |
| Evaluation Criteria | Defines how responses are assessed. Criteria include evaluation context and one or more scoring guidelines. |
| Scoring Guideline | A rule that defines how a response is scored on a numeric scale. Each guideline produces a separate score. |
| LLM Evaluator | The large language model used to evaluate and score AI Employee responses. |
| Evaluation Results | Row-level outputs containing scores. Results can be viewed in the UI or downloaded as CSV. |
Permissions
Only users with the AI Employee Manager or AI Employee Admin role can view and run evaluations in the Evaluations tab.
Creating an Evaluation
Evaluations are created from the Evaluations tab of a chat-triggered AI Employee by clicking New evaluation.
Evaluation Setup
Dataset Selection
You can either:
- Select an existing dataset (previously uploaded), or
- Upload a new dataset.
Input Mapping
Each evaluation requires mapping one dataset column as the user query. Values from this column are used to trigger the AI Employee for evaluation.
Evaluation Criteria
You can either:
- Use an existing default evaluation criteria or previously saved criteria, or
- Create a new evaluation criteria.
Evaluation Context
The evaluation context defines the instructions and variables used to guide scoring.
Pills: Pills can be selected from the left panel and inserted into the evaluation context. Each pill represents a variable and is replaced with its corresponding value before being sent to the LLM evaluator.
Scoring Guidelines
At least one scoring guideline is required per evaluation criteria. Each scoring guideline includes:
- Name (required)
- Description (optional)
- Scale -- An integer range for the scoring scale (e.g., 1 to 3, 1 to 5, or custom ranges).
Each numeric value within the selected scale must have a clearly defined criterion to ensure consistent and reliable scoring.
LLM Evaluator Selection
Each evaluation requires selecting an LLM evaluator. The selected model is used as a judge to score all dataset rows. The dropdown displays all available models, including any custom models added on the AI Employee's configuration page.
Running Evaluations
- Configure the dataset and evaluation criteria.
- Click Run Evaluation to start.
Monitoring Progress
- Each dataset row displays an execution status.
- Rows marked Success indicate completed scoring.
Results and Metrics
Evaluation Results
After completion:
- Full results are available in the evaluation view.
- Results can be downloaded as a CSV file.
Metrics
For each scoring guideline, the following metrics are generated:
- Highest score
- Lowest score
- Average score
- Score distribution
Notes
Notes can be added to evaluations for documentation and future reference.
Past Evaluations
The Evaluations tab lists all historical evaluations for the AI Employee, making it easy to compare performance over time.