Building a Test Dataset

A test dataset is the set of test cases an evaluation run plays against your orchestrator. You describe what the test cases should cover and add documents if you have them, and Ema writes the test cases for you. This guide walks through building a dataset, reading what Ema wrote, and changing it afterwards.

If you have not read it yet, Evaluating Orchestrator AI Employees explains how test datasets and evaluation runs fit together.

Before you start

The orchestrator must be published. Ema writes the test cases against the published version, including its connected AI Employees and knowledge bases.
You need the Builder or Admin role on the orchestrator. See Who can do what.

Step 1: Create the dataset

Open the orchestrator and select the Training tab.
Select Test datasets, then Build a dataset.
In New test dataset, enter a Name and select Create dataset. You can rename it at any time.

The dataset opens as a draft, ready for you to describe.

Step 2: Write the goal

The Goal says what the test cases should cover, and it is required. A goal like "Test PTO requests" gets you generic test cases. A goal that names the situations, the edge cases, and the rules the orchestrator must follow gets you test cases that look like your real work. A strong goal names:

the requests you want tested, including the unusual ones
the rules and limits the orchestrator must follow
what a good outcome looks like

For example:

Test how this orchestrator handles employee PTO requests end to end: new requests, changes and cancellations, requests that overlap a public holiday, and requests with too little balance left. It should follow the uploaded SOPs, act only on the requester's own record, and confirm before making changes. A good outcome is the request filed correctly, or a clear reason why it cannot be.

Let Autopilot write the goal. Open Autopilot and ask it to write the goal for this dataset. It asks you what it needs to know, such as the situations to cover and how people usually talk to the orchestrator, then writes the goal and shows it to you before saving it.

To start from what the orchestrator is already set up to do, select Use this goal under This AI Employee's published configuration. The goal is filled in from the published orchestrator, and you can edit it from there.

What you write here is only used to build the dataset. It does not change the live orchestrator.

Step 3: Add sources (optional)

Test cases are always grounded in the orchestrator's knowledge bases, so their expected answers match what the orchestrator knows. Under Sources, you can add documents to ground them in more:

Reference documents: SOPs, policies, and reference docs, so the test cases follow your team's rules and guardrails.
Past examples: past tickets, chat logs, and sample records, so the test cases mirror how the work really happens.

Drag files onto a card, or select Browse files. Supported formats are PDF, DOCX, PPTX, XLSX, HTML, TXT, MD, JSON, and CSV, up to 25 MB each. Removing a document takes effect straight away, without asking you to confirm.

Step 4: Generate the test cases

Under Test cases, enter how many to generate, from 1 to 100. The default is 20. More test cases take longer to generate and to run, and you can add more later.
Select Generate dataset. The button stays unavailable until you have written a goal.
In Generate this dataset?, select Generate now, or Not yet to keep editing.

While the page shows "Generating your dataset…", Ema reads your sources and plans the test cases first, which can take a few minutes, and then the test cases appear as they are ready. Generation keeps running if you leave, and you can come back to it from Test datasets at any time.

The goal and documents lock while generation runs, and stay locked once the first test case is written. You can still rename the dataset.

Nothing touches your live systems. Tools are simulated while test cases are written: no records are created, no messages are sent, and no integrations are called.

Step 5: Read the test cases

The dataset lists each test case with its Summary, Opening message, and Expected output. Select a test case to open it. Each one has four parts:

Part	What it tells you
Opening message	What the simulated user says first.
User behavior	How the simulated user acts for the whole conversation: Why they are asking, How they behave, What they know, and What they do not know.
Simulated environment	What Ema makes true while the test case runs: What is true, and How tools will behave. None of it touches your live systems.
Expected outcome	What the conversation is graded against: what The answer must say, the Tools that must be used, and the AI Employees that must be involved. When the orchestrator should not hand the request to anyone, this reads "None. The orchestrator handles this one alone."

Use Previous test case and Next test case to step through them. Read a handful before you run the dataset. Each conversation is graded against the expected outcome, so a test case that expects the wrong thing marks an orchestrator that did the right thing as failing. Delete any test case like that.

Step 6: Add or delete test cases

Test cases cannot be edited or regenerated. You shape a dataset by adding test cases and deleting the ones you do not want.

Add test cases. Select Add test cases and enter How many, from 1 to 100. Under What should these cover?, describe what the existing test cases are missing, for example contractors and interns, whose equipment rules differ from full-time joiners. The description can be up to 500 characters. The new test cases join the end of the dataset and follow the same goal and sources. Your existing test cases are untouched.

Delete test cases. Delete a single test case from its row, or select several and choose Delete selected. You are asked to confirm. The rest of the dataset is untouched.

Clone, rename, or delete a dataset

Open the ⋮ menu on a dataset's row in Test datasets:

Clone makes a working copy you can change freely: all the test cases in the same order, the goal, and the documents it was built from. The original is untouched.
Rename changes the dataset's name.
Delete removes the dataset, its test cases, and the documents uploaded to it. This cannot be undone. A dataset that has been used in an evaluation run cannot be deleted.

When a dataset locks

The first evaluation run of a dataset locks it for good. The dataset then shows "This dataset has been used in an evaluation", and its test cases are kept exactly as they are so the reports that graded them stay readable. You can no longer add or delete test cases, but you can still rename it. To keep working on it, Clone it: the copy starts unlocked.

While an evaluation run is using a dataset, nothing about it can change, including its name, until every active run finishes.

Ask Autopilot about a dataset

Once a dataset has been generated, select Explain this dataset to have Autopilot explain what its test cases cover. You can also ask Autopilot to build a dataset for you: create it, write the goal, generate the test cases, add or delete test cases, and clone it. Autopilot cannot delete a dataset, edit a test case, or upload files from your computer.

Troubleshooting

Symptom	Cause and fix
Generate dataset is unavailable ("Write a goal first.")	The dataset has no goal yet. Write one under Goal.
"This AI Employee has not been published"	Test cases are written against the published orchestrator. Publish it, then generate again.
"This dataset has no goal"	The goal was empty when generation started. Write a goal and try again.
"This dataset has already been generated"	Each dataset is generated once. To get more test cases, use Add test cases. To start over, build a new dataset.
"Test cases cannot be generated for this AI Employee"	Ema could not read this AI Employee as a published orchestrator, for example because it was changed or deleted while the page was open. Reload the page and try again.
"Generation could not be set up for this workspace"	To write test cases, Ema sets up hidden copies of the orchestrator, and that step failed. Try again. If it keeps happening, ask an administrator to check that your workspace can create AI Employees.
"No test cases could be generated"	Your goal and sources are still editable. Rewrite the goal so it names the situations you want tested, then generate again. If the message says the run stopped for a reason on Ema's side, generate again without changes.
"The test cases we tried to add did not generate"	Your existing test cases are unchanged. Select Add test cases again with a narrower description under What should these cover?
"No test cases were added"	The dataset changed after the page last read it. Reload the page and try again.
A document shows "Over 25 MB", "Unsupported format", or "Unreadable file"	Use a file under 25 MB in one of the supported formats. If a supported file cannot be read, try a different copy of it.
A document shows "Duplicate name"	A file with that name is already in the dataset. Rename the file to add a new version.
"Document limit reached"	The dataset already holds 100 documents. Remove one to add another.
You cannot add or delete test cases ("This dataset has been used in an evaluation")	The dataset locked when its first evaluation run started. Clone it and change the copy.

What's next

Running an Evaluation: run this dataset against the orchestrator and read the results.
Evaluating Orchestrator AI Employees: how evaluations work, and who can do what.