Building a Test Dataset
A test dataset is the set of test cases an evaluation run plays against your AI Employee. You describe what the test cases should cover and add documents if you have them, and Ema writes the test cases for you. This guide walks through building a dataset, reading what Ema wrote, and changing it afterwards.
If you have not read it yet, Evaluating Orchestrator AI Employees or Evaluating Chat AI Employees explains how test datasets and evaluation runs fit together for that type.
Before you start
- The AI Employee must be published. Ema writes the test cases against the published version, including its knowledge bases and, for an orchestrator, its connected AI Employees.
- You need the Builder or Admin role on the AI Employee. See Who can do what.
Step 1: Create the dataset
- Open the AI Employee and select the Training tab.
- Select Test datasets, then Build a dataset.
- In New test dataset, enter a Name and select Create dataset. You can rename it at any time.
The dataset opens as a draft, ready for you to describe.
Step 2: Write the goal
The Goal says what the test cases should cover, and it is required. A goal like "Test PTO requests" gets you generic test cases. A goal that names the situations, the edge cases, and the rules the AI Employee must follow gets you test cases that look like your real work. A strong goal names:
- the requests you want tested, including the unusual ones
- the rules and limits the AI Employee must follow
- what a good outcome looks like
For example:
Test how this AI Employee handles employee PTO requests end to end: new requests, changes and cancellations, requests that overlap a public holiday, and requests with too little balance left. It should follow the uploaded SOPs, act only on the requester's own record, and confirm before making changes. A good outcome is the request filed correctly, or a clear reason why it cannot be.
Let Autopilot write the goal. Open Autopilot and ask it to write the goal for this dataset. It asks you what it needs to know, such as the situations to cover and how people usually talk to the AI Employee, then writes the goal and shows it to you before saving it.
To start from what the AI Employee is already set up to do, select Use this goal under the suggestion. For an orchestrator, the suggestion is This AI Employee's published configuration and comes from its published Instructions. For a Chat AI Employee, it is This AI Employee's published summary, and it appears only while the summary describes the published version. You can edit the goal from there.
What you write here is only used to build the dataset. It does not change the live AI Employee.
Step 3: Add sources (optional)
Test cases are always grounded in the AI Employee's knowledge bases, so their expected answers match what it knows. Under Sources, you can add documents to ground them in more:
- Reference documents: SOPs, policies, and reference docs, so the test cases follow your team's rules and guardrails.
- Past examples: past tickets, chat logs, and sample records, so the test cases mirror how the work really happens.
Drag files onto a card, or select Browse files. Supported formats are PDF, DOCX, PPTX, XLSX, HTML, TXT, MD, JSON, and CSV, up to 25 MB each. Removing a document takes effect straight away, without asking you to confirm.
Step 4: Generate the test cases
- Under Test cases, enter how many to generate, from 1 to 100. The default is 20. More test cases take longer to generate and to run, and you can add more later.
- Select Generate dataset. The button stays unavailable until you have written a goal.
- In Generate this dataset?, select Generate now, or Not yet to keep editing.
While the page shows "Generating your dataset…", Ema reads your sources and plans the test cases first, which can take a few minutes, and then the test cases appear as they are ready. Generation keeps running if you leave, and you can come back to it from Test datasets at any time.
The goal and documents lock while generation runs, and stay locked once the first test case is written. You can still rename the dataset.
Nothing touches your live systems. Tools are simulated while test cases are written: no records are created, no messages are sent, and no integrations are called.
Sample from production sessions
Orchestrators only, where enabled. Sampling from sessions is available for orchestrator AI Employees, and only when it is enabled for your workspace. Where it is not, the Sample from sessions option does not appear and Ema generates every test case from your goal and sources.
Instead of having Ema write every test case, you can have it draw them from real sessions in the AI Employee's Audit tab. Under Sample from sessions on the draft page, tick Use sessions that started and pick the days those sessions started in: up to 90 days, ending today at the latest. The range starts on the last 30 days, and the picker offers presets such as Last 7 days, Last 90 days, This month, and Last month.
Ema reads the sessions that started in that range and keeps the ones that match your goal and went well, meaning every request the user made was met. Each one becomes a test case marked Sampled from a session, whose opening message is what the user typed. Sessions rated thumbs-down, sessions that ran on an earlier published version, and repeats of an opening message already picked are left out. With sampling on, the number under Test cases is how many you want in total: Ema uses matching sessions first and, if too few match, generates the rest from your goal and sources.
Before you generate, Ema checks the AI Employee's sessions:
- If no one has used the AI Employee yet, the checkbox is unavailable and says "No one has used this AI Employee yet, so there are no sessions to sample."
- If no session started in the range you picked, "No sessions started between [start] and [end]. Pick other dates." appears under the dates, and Generate dataset stays unavailable until you pick other dates.
Generate this dataset? repeats the range and how many matching sessions Ema keeps. If the range is refused when generation starts, for example because it ends after today or spans more than 90 days, the reason appears in the dialog so you can change the dates without losing your other settings.
A sampled generation can take a few hours while Ema reads sessions. Until the first test case is written, the counter shows how many matching sessions have been found, for example "4 of 20 test cases found". Once writing starts, it counts the test cases written against the number you asked for. The dataset's summary then shows Where the test cases came from: how many were Sampled from sessions, Generated by Ema, and Added.
Ask Autopilot to use real sessions. Where sampling is enabled, you can also ask Autopilot to build an orchestrator's test cases from its real sessions, for a new dataset or added to an existing one. It proposes the sampled run and waits for your go-ahead before starting it.
Step 5: Read the test cases
The dataset lists each test case with its Summary, Opening message, and Expected output. Select a test case to open it. Below its Summary, each one has four parts:
| Part | What it tells you |
|---|---|
| Opening message | What the simulated user says first. |
| Expected outcome | What the conversation is graded against: what The answer must say, and the Tools that must be used (shown when the test case expects any). For an orchestrator, it also lists the AI Employees that must be involved; when the orchestrator should not hand the request to anyone, this reads "None. This AI Employee handles this one alone." |
| Simulated environment | What Ema makes true while the test case runs: What is true, and How tools will behave. None of it touches your live systems. |
| User behavior | How the simulated user acts for the whole conversation: Why they are asking, How they behave, What they know, and What they do not know. |
A test case can also show a Test case date at the top of Simulated environment. It is set automatically and cannot be edited: a generated test case takes the day it was generated, one sampled from a session takes the day that session started, and one added with Add test cases takes the day it was added, all in UTC. Editing or cloning a test case keeps its date. An evaluation run treats that day as today, so a test case written around a particular date is graded the same way whenever it runs. See Test case dates.
Use Previous test case and Next test case to step through them. Read a handful before you run the dataset. Each conversation is graded against the expected outcome, so a test case that expects the wrong thing marks an AI Employee that did the right thing as failing. Fix its expected answers, or delete it.
Step 6: Change test cases
You shape a dataset by editing expected answers, adding test cases, and deleting the ones you do not want. Every change saves a new version of the dataset. You can change a dataset at any time, even while an evaluation run is using it: the run keeps grading the version it started with.
Edit expected answers. Open a test case and select Edit under Expected outcome. Reword an answer, select Add an expected answer, or remove one, then select Save changes. A test case needs at least one answer and can have up to 50, each up to 1,000 characters. Only the expected answers can be edited. To change the rest of a test case, delete it and use Add test cases to describe what you want instead.
Add test cases. Select Add test cases and enter How many, from 1 to 100. Under What should these cover?, you can describe what the existing test cases are missing, for example contractors and interns, whose equipment rules differ from full-time joiners. The description is optional and can be up to 500 characters; without one, Ema writes test cases for what the existing ones leave uncovered. The new test cases join the end of the dataset and follow the same goal and sources. Your existing test cases are untouched.
For an orchestrator where sampling is enabled, Add test cases also has Sample from sessions. Tick Use sessions that started and pick up to 90 days. Ema turns matching sessions into test cases first, skipping any session this dataset has already used, including in past versions and in deleted test cases, and generates the rest if too few are left. A description under What should these cover? narrows which sessions are picked. If the dataset was first built from sessions, the option opens ticked on the same days.
Delete test cases. Delete a single test case from its row, or select several and choose Delete selected. You are asked to confirm. A deleted test case leaves the current version only: past versions and the evaluation runs that graded it keep it.
Download a dataset
Select Download at the top of a dataset to save its test cases as a CSV file, one row per test case. The file is UTF-8 encoded, so Excel shows accented characters correctly. Its name combines the AI Employee's name, the dataset's name, the version, and the date, for example employee-help_pto-requests_v4_2026-11-05.csv. To download an earlier version, open it from Version history and select Download there.
| Column | Contents |
|---|---|
# | The test case's number, as shown in the dataset. |
input | The opening message. |
persona.reason, persona.instructions, persona.known_info, persona.unknown_info | The User behavior fields. |
world_facts.facts, world_facts.tool_behaviors | The Simulated environment. |
expected.aies | The AI Employees that must be involved. Orchestrator datasets only. |
expected.tools, expected.answers | The Tools that must be used, and what the answer must say. |
summary | The test case's summary. |
test_case_id | An ID that stays the same across versions, so you can match a row to the same test case in a run report. |
test_case_source, source_session_id | Where the test case came from (sampled, synthetic, or added), and the session it was sampled from, if any. |
Lists are written as JSON, one item per line. A cell that would start like a spreadsheet formula is prefixed with an apostrophe, so it opens as text.
Download is unavailable while test cases are still being generated ("Available when generation finishes"), while a cloned dataset is still copying ("Available when the copy finishes"), and before the dataset has any test cases ("Generate test cases first").
Clone, rename, or delete a dataset
Open the ⋮ menu on a dataset's row in Test datasets:
- Clone makes a new dataset from the current test cases, in the same order, with the goal and documents it was built from. The copy starts at v1 with no history. The original is untouched. Use it to keep a dataset as it stands and change a copy.
- Rename changes the dataset's name.
- Delete removes the dataset, its test cases, and the documents uploaded to it. This cannot be undone. A dataset that has evaluation runs cannot be deleted, because their reports depend on it.
Versions and history
Every change to a dataset's test cases saves a new version: v1 when it is generated, then a new one each time you add, edit, or delete test cases. The chip next to the dataset's name, for example v4, shows the current version.
Select the chip to open Version history. Each entry says what changed, such as "Edited 1 test case" or "Deleted 2 test cases", who changed it, and when. The current version is marked Current. Select an earlier entry to open that version. A past version is Read only. It shows the test cases as they were, with a mark on each one added, edited, or deleted in that version, and deleted test cases stay in the list, greyed out. To return, select the dataset's name in the breadcrumb.
To see how one test case changed, open it and select the History tab. Each entry names the version it changed in. Edits to expected answers show which answers were added, removed, or reworded.
The goal and documents are the same in every version. You cannot restore an earlier version or run an evaluation against one. A new run always grades the current version.
Ask Autopilot about a dataset
Once a dataset has been generated, select Explain this dataset to have Autopilot explain what its test cases cover. You can also ask Autopilot to build a dataset for you: create it, write the goal, generate the test cases, add test cases, edit expected answers, delete test cases, clone it, and read its version history. Autopilot cannot delete a dataset, restore an earlier version, or upload files from your computer.
When you ask Autopilot to check an AI Employee you are building or changing, it can test it with a development dataset of its own, named AI Employee name (development). It writes about 30 test cases aimed at what you built or changed, runs them against the published version, and reads the results with you.
Troubleshooting
| Symptom | Cause and fix |
|---|---|
| Generate dataset is unavailable ("Write a goal first.") | The dataset has no goal yet. Write one under Goal. |
| "This AI Employee has not been published" | Test cases are written against the published AI Employee. Publish it, then generate again. |
| "This dataset has no goal" | The goal was empty when generation started. Write a goal and try again. |
| "This dataset has already been generated" | Each dataset is generated once. To get more test cases, use Add test cases. To start over, build a new dataset. |
| "Test cases cannot be generated for this AI Employee" | Ema could not read this AI Employee's published version, for example because it was changed or deleted while the page was open. Reload the page and try again. |
| "Generation could not be set up for this workspace" | To write test cases, Ema sets up hidden copies of the AI Employee, and that step failed. Try again. If it keeps happening, ask an administrator to check that your workspace can create AI Employees. |
| "No test cases could be generated" | Your goal and sources are still editable. Rewrite the goal so it names the situations you want tested, then generate again. If the message says the run stopped for a reason on Ema's side, generate again without changes. |
| "The test cases we tried to add did not generate" | Your existing test cases are unchanged. Select Add test cases again with a narrower description under What should these cover? |
| "No test cases were added" | The dataset changed after the page last read it. Reload the page and try again. |
| A document shows "Over 25 MB", "Unsupported format", or "Unreadable file" | Use a file under 25 MB in one of the supported formats. If a supported file cannot be read, try a different copy of it. |
| A document shows "Duplicate name" | A file with that name is already in the dataset. Rename the file to add a new version. |
| "Document limit reached" | The dataset already holds 100 documents. Remove one to add another. |
| Previous test case and Next test case are missing | The dataset is still generating, and test cases are still being added. Open each test case from the list. The buttons come back once generation finishes. |
| Delete is unavailable ("This dataset has evaluation runs, and their reports depend on it") | A dataset that has been run cannot be deleted, so its reports stay readable. To start fresh, Clone it. |
| "Sessions can only be used from [date] or earlier. Pick an earlier end date and try again." | The range you picked under Sample from sessions ends after today. Pick an end date of today or earlier. |
| "Sessions can be read from up to 90 days at a time. Pick a shorter range and try again." | The range you picked spans more than 90 days. Shorten it and try again. |
| "No sessions started between [start] and [end]. Pick other dates." | No session of this AI Employee started in the range you picked. Pick other dates. |
| "Sampling from sessions is not turned on for this workspace. Untick Sample from sessions to generate every test case." | Sampling from sessions is not enabled for your workspace. Untick Sample from sessions to generate every test case from your goal and sources. |
What's next
- Running an Evaluation: run this dataset against the AI Employee and read the results.
- Evaluating Orchestrator AI Employees and Evaluating Chat AI Employees: how evaluations work for each type, and who can do what.