> Source: https://builder.ema.ai/v2/testing-operations/running-an-evaluation > Title: Running an Evaluation # Running an Evaluation An evaluation run plays every test case in a test dataset as a simulated conversation with your orchestrator, then grades each conversation on its **Response** and its **Plan**. This guide covers starting a run, reading its results, and using them to improve the orchestrator. You need a test dataset that has finished generating. If you do not have one yet, start with [Building a Test Dataset](/builder/v2/testing-operations/building-a-test-dataset). For how the pieces fit together, see [Evaluating Orchestrator AI Employees](/builder/v2/testing-operations/evaluating-orchestrators). ## Before you start - The orchestrator must be **published**. A run always tests the version published right now. - You need the **Manager**, **Builder**, or **Admin** role on the orchestrator. See [Who can do what](/builder/v2/testing-operations/evaluating-orchestrators#who-can-do-what). ## Step 1: Start a run 1. Open the orchestrator's **Training** tab. **Evaluation runs** opens first. 2. Select **New evaluation run**. You can also open a dataset from **Test datasets** and select **Run this dataset**. 3. Enter a **Name** that says what the run is for. It is how you pick the run out of the list later. 4. Choose the **Dataset** to score against. 5. Check **What this run will do**. It confirms that connections to your other apps are simulated and names the version the run tests, for example "Test Employee Help v4, the version published right now." 6. Select **Start run**. > [WARNING] > **The first run locks the dataset.** Once a dataset's first evaluation run starts, you can no longer add or delete its test cases, so every report that graded them stays readable. To change the dataset after that, clone it. See [When a dataset locks](/builder/v2/testing-operations/building-a-test-dataset#when-a-dataset-locks). An orchestrator can only have a limited number of evaluation runs going at once. If it has reached that limit, you see "This AI Employee is already running as many as it can". Start the new run when one of the active runs finishes. ## Step 2: Follow the run The run page shows "Evaluation run in progress…" and a count such as "12 of 20 test cases". The run keeps going if you leave, and you can come back to it from **Evaluation runs** at any time. **About this run** shows the dataset it is **Scored against**, the **AI Employee version** it tests, when it **Started**, and who it was **Run by**. To end a run early, select **Stop run** and confirm. Test cases already graded keep their results, and conversations already underway are still graded. A run shows one of these statuses: Status What it means **Running** Conversations are being held and graded. **Stopping** Someone stopped the run. Conversations already underway are still being graded. The run then ends as **Cancelled** if some test cases never started, or with one of the other statuses if every test case had already started. **Completed** The run finished and every test case was graded. **Partly graded** Every test case ran, but some stopped on an error before they were graded. **Retry** grades them. See [Troubleshooting](#troubleshooting). **Failed** The run could not grade its test cases. **Cancelled** The run was stopped before some test cases started. **Run the remaining** runs them. ## Step 3: Read the results Two cards at the top of the run page give the scores: - **Response accuracy**: how often the orchestrator produced the correct answer. - **Plan accuracy**: how often it followed the intended steps, tools, and routing to reach an answer. Each card also reads, for example, "15 of 18 graded test cases passed". The rates count graded test cases only. The two checks are graded separately, so a test case can pass one and fail the other. Below the cards, **Test cases** lists every test case with a **Response** mark and a **Plan** mark. To focus on one check, filter the list to **Response passed**, **Response failed**, **Plan passed**, or **Plan failed**. **Failed** shows every test case that failed at least one check, and **Passed** the ones that passed both. A test case that was not graded shows **Not graded** and cannot be opened. Select a test case to open it: - The **Verdict** tab shows a **Response** card and a **Plan** card, each marked **Passed** or **Failed**, with the reasons: **Problems found** when it failed, **Why this passed** when it passed. The test case itself follows, so you can compare what happened with what was expected. - The **Conversation** tab shows the whole simulated conversation. Where the orchestrator handed work to a connected AI Employee, a "Delegated to" card shows what that AI Employee did. If your role lets you see detailed logs, the card has a **View detailed logs** link that opens that AI Employee's [debug logs](/builder/v2/testing-operations/debug-logs). Select **Copy link** to share the test case with a teammate. ## Step 4: Improve and run again 1. Filter the results to **Failed** and open each failed test case. Read **Problems found** on the **Verdict** tab, then open the **Conversation** tab to see what the orchestrator did. Where it handed work to a connected AI Employee, select **View detailed logs** to see what that AI Employee did. 2. Group the failures that share a cause. Several test cases often fail for the same reason, for example the orchestrator sending billing questions to the wrong AI Employee, or a connected AI Employee leaving out the same detail. Each group needs one fix, not one per test case. 3. Decide where each group's fix belongs, make it, and **Publish**: - If the orchestrator chose the wrong AI Employee, sent it the wrong request, or answered badly, change the orchestrator's Instructions on the **Configuration** tab. - If a connected AI Employee got a good request and still did the wrong thing, fix and publish that AI Employee. 4. Open the run's ⋮ menu and select **Re-evaluate**. A new run starts on the same dataset, against the version you just published. > [TIP] > **Let Autopilot do the grouping.** Select **Ask Autopilot** > **Improve** _orchestrator name_. Autopilot reads the failed conversations, including the detailed logs of the connected AI Employees involved, groups the failures by cause, and proposes one change per group to the orchestrator, the connected AI Employees, or both. It tells you which AI Employee each change affects and asks before changing anything. Once you confirm, it can publish the changes and run the evaluation again. Every run stays in the **Evaluation runs** list, which shows each run's version, **Response accuracy**, and **Plan accuracy**, so you can see whether a change helped. To rename a run, use **Rename** in its ⋮ menu. ## Ask Autopilot about a run - **Explain these results** on the run page asks [Autopilot](/builder/v2/autopilot) to explain the run's results. - **Explain this result** in an open test case asks about that one test case. - **Ask Autopilot** > **Improve** _orchestrator name_ proposes fixes from the failed test cases, as described in [Step 4](#step-4-improve-and-run-again). You can also ask Autopilot to start, stop, or retry a run, and to read its results. Autopilot cannot rename a run. ## Troubleshooting Symptom Cause and fix "Publish this AI Employee to use evaluation runs" The orchestrator has not been published. Publish it from the **Configuration** tab. Earlier runs stay available to read. "Build a dataset first", or no dataset to choose A run needs a dataset that has finished generating. See [Building a Test Dataset](/builder/v2/testing-operations/building-a-test-dataset). "This AI Employee is already running as many as it can" The orchestrator is at its limit of evaluation runs going at once. Start this run when one of them finishes. "This dataset changed and cannot be run right now" The dataset changed after you opened the dialog. Review it and try again. "You no longer have permission to start evaluation runs" Your role on the orchestrator changed. Ask an Admin for the **Manager** role or higher. Some test cases show **Not graded** They stopped on an error before they were scored, or the run was stopped before it reached them. The banner above the results says how many. Select **Retry** to grade the ones that stopped on an error, or **Run the remaining** to finish a cancelled run. The banner offers only **Start a new run** The orchestrator has been published again since the run started. Finishing the old run would mix two versions in one score, so start a new run against the current version instead. **Response** passed but **Plan** failed The orchestrator gave the right answer by a route the test case did not expect, such as without the AI Employee it should have involved. Read **Problems found** on the **Plan** card of the **Verdict** tab. ## What's next - [Building a Test Dataset](/builder/v2/testing-operations/building-a-test-dataset): add test cases where the results show gaps, or clone a locked dataset. - [Debug Logs](/builder/v2/testing-operations/debug-logs): read a connected AI Employee's run step by step. - [Writing effective Instructions](/builder/v2/builder-guides/writing-instructions): Instructions that route requests reliably. - [Evaluating Orchestrator AI Employees](/builder/v2/testing-operations/evaluating-orchestrators): how evaluations work, and who can do what.