Evaluating Orchestrator AI Employees
Chatting with an orchestrator shows you how it handles the few requests you think to try. An evaluation tests it against dozens of realistic requests at once and shows you which ones it got wrong, so you can fix them and check that the fix worked.
An evaluation has two parts:
- A test dataset is a set of test cases that Ema writes for you from a goal you describe and the documents you add.
- An evaluation run plays every test case in a dataset as a simulated conversation with the orchestrator, then grades each one.
You do both on the orchestrator's Training tab. This page explains how an evaluation works and who can do what. Each part then has a guide of its own.
Orchestrators only. The Training tab appears on Orchestrator AI Employees. To test a Chat, Voice, or Dashboard AI Employee, see Testing AI Employees.
How an evaluation works
- Build a test dataset. Describe what the test cases should cover, add reference documents or past examples if you have them, and choose how many test cases you want. Ema writes them.
- Start an evaluation run. Pick a dataset. Ema holds one simulated conversation per test case with the version of the orchestrator that is published right now, and grades each conversation.
- Improve and run again. See which test cases failed and why, fix the orchestrator or the AI Employees it hands work to, publish, and run the same dataset again.
Nothing in an evaluation touches your live systems. Connections to your other apps are simulated, so no records are created, no messages are sent, and no tool calls reach those apps. When the orchestrator hands work to a connected AI Employee, that AI Employee really runs, but its connections to other apps are simulated too.
Key terms
Test case. One request to the orchestrator, written as a scenario: what a simulated user says first, how they behave, what is true while the conversation runs, and the outcome the conversation is graded against.
Response. Whether the orchestrator's answer was correct. Response accuracy is how often it was.
Plan. Whether the orchestrator took the intended route to its answer: the right steps, the right tools, and the right connected AI Employees. Plan accuracy is how often it did.
Each test case is graded on both checks separately, so it can pass one and fail the other. Both rates count graded test cases only, so a test case that could not be graded is left out of them.
Who can do what
What you can do depends on your role on the orchestrator, which is set on its Permissions tab.
| Action | Admin | Builder | Manager | Guest |
|---|---|---|---|---|
| View test datasets, evaluation runs, and their results | Yes | Yes | Yes | Yes |
| Start, stop, rename, and re-evaluate evaluation runs, and retry test cases that were not graded | Yes | Yes | Yes | No |
| Build, generate, rename, clone, and delete test datasets, and add or delete test cases | Yes | Yes | No | No |
These are the four roles the Permissions tab offers. A Guest can use the orchestrator and read its evaluations; people who can use it because it is shared with everyone in your organization are Guests. A Manager can also run evaluations. A Builder can also build and change test datasets. An Admin can do everything.
Building a test dataset
A test dataset starts from a goal: a few sentences on what the test cases should cover, for example how the orchestrator handles time-off requests from start to finish. You can add reference documents, such as SOPs and policies, so the test cases follow your team's rules, and past examples, such as old tickets or chat logs, so they look like real work. Ema also draws on the orchestrator's knowledge bases.
Each test case sets out a simulated user, what is true while the conversation runs, and the expected outcome. Once written, test cases cannot be edited, but you can add more and delete the ones you do not want.
Running an evaluation
An evaluation run grades every test case in a dataset against the published orchestrator. The run page shows Response accuracy and Plan accuracy, a list of test cases you can filter to the failures, and for each test case the problems found and the full conversation, including what each connected AI Employee did. From there you group the failures by cause, fix and publish, and Re-evaluate on the same dataset.
What's next
- Building a Test Dataset: the goal, sources, generating, and reading and changing test cases.
- Running an Evaluation: starting a run, reading Response and Plan results, and improving the orchestrator.
- Creating an Orchestrator AI Employee: build and publish the orchestrator you want to evaluate.
- Orchestrator AI Employees: how orchestrators hand work to other AI Employees.