> Source: https://builder.ema.ai/v2/testing-operations/evaluating-chat-ai-employees > Title: Evaluating Chat AI Employees # Evaluating Chat AI Employees Chatting with a Chat AI Employee shows you how it handles the few requests you think to try. An **evaluation** tests it against dozens of realistic requests at once and shows you which ones it got wrong, so you can fix them and check that the fix worked. An evaluation has two parts: - A **test dataset** is a set of test cases that Ema writes for you from a goal you describe and the documents you add. - An **evaluation run** plays every test case in a dataset as a simulated conversation with the AI Employee, then grades each one. You do both on the AI Employee's **Training** tab. This page explains how an evaluation of a Chat AI Employee works and who can do what. Building a dataset and running an evaluation each have a guide of their own, shared with [Orchestrator AI Employees](/builder/v2/testing-operations/evaluating-orchestrators). > [INFO] > **Chat AI Employees.** This page covers Chat AI Employees, which have a **Training** tab like orchestrators. To test a Voice or Dashboard AI Employee, see [Testing AI Employees](/builder/v2/testing-operations/testing-ai-employees). ## How an evaluation works 1. **Build a test dataset.** Describe what the test cases should cover, add reference documents or past examples if you have them, and choose how many test cases you want. Ema writes them. 2. **Start an evaluation run.** Pick a dataset. Ema holds one simulated conversation per test case with the version of the AI Employee that is published right now, and grades each conversation. 3. **Improve and run again.** See which test cases failed and why, fix the AI Employee's workflow, publish, and run the same dataset again. Nothing in an evaluation touches your live systems. The AI Employee's workflow really runs, but its connections to your other apps are simulated, so no records are created, no messages are sent, and no tool calls reach those apps. ## Key terms **Test case.** One request to the AI Employee, written as a scenario: what a simulated user says first, how they behave, what is true while the conversation runs, and the outcome the conversation is graded against. **Response.** Whether the AI Employee's answer was correct. **Response accuracy** is how often it was. **Plan.** Whether the AI Employee chose the right tools to reach its answer. When a test case expects no tools and the AI Employee uses none, Plan passes. **Plan accuracy** is how often it chose the right tools. Each test case is graded on both checks separately, so it can pass one and fail the other. Both rates count graded test cases only, so a test case that could not be graded is left out of them. ## How the conversation is simulated Each test case is played as a chat with the AI Employee. A simulated user opens with the test case's opening message and keeps talking the way the test case says they behave, until the request is done or the conversation reaches its limit. When the workflow stops to ask the user for something, the simulated user answers it the way the person in the test case would: - **Approvals.** The simulated user approves or rejects, with a comment when they have one. - **Questions and forms.** The simulated user answers the question or fills in the form. - **Requests to connect an account.** These are not answered. The simulated user cannot sign in to another app, so the conversation ends there and is graded on what happened up to that point. Evaluation conversations do not appear on the AI Employee's **Chat** tab. You read them in the evaluation run's results. ## Who can do what What you can do depends on your role on the AI Employee, which is set on its **Permissions** tab. Action Admin Builder Manager Guest View test datasets, evaluation runs, and their results Yes Yes Yes Yes Start, stop, rename, and re-evaluate evaluation runs, and retry test cases that were not graded Yes Yes Yes No Build, generate, rename, clone, and delete test datasets, and add or delete test cases Yes Yes No No These are the four roles the **Permissions** tab offers. A **Guest** can use the AI Employee and read its evaluations; people who can use it because it is shared with everyone in your organization are Guests. A **Manager** can also run evaluations. A **Builder** can also build and change test datasets. An **Admin** can do everything. ## Building a test dataset A test dataset starts from a **goal**: a few sentences on what the test cases should cover, for example how the AI Employee handles time-off requests from start to finish. To start from what the AI Employee is already set up to do, use the goal suggested under **This AI Employee's published summary**. You can add reference documents, such as SOPs and policies, so the test cases follow your team's rules, and past examples, such as old tickets or chat logs, so they look like real work. Ema also draws on the AI Employee's knowledge bases. Each test case sets out a simulated user, what is true while the conversation runs, and the expected outcome: what **The answer must say**, and the **Tools that must be used** when the request needs any. [Build a test dataset →](/builder/v2/testing-operations/building-a-test-dataset) ## Running an evaluation An evaluation run grades every test case in a dataset against the published AI Employee. The run page shows **Response accuracy** and **Plan accuracy**, a list of test cases you can filter to the failures, and for each test case the problems found and the full conversation. The **Conversation** tab reads the way the **Chat** tab draws a conversation. Each approval or form the workflow raised shows how the simulated user answered it, for example **Approved** or **Rejected** with their comment. A request the simulated user did not answer reads "Not answered before the conversation ended.", and a request to connect an account reads "Asked to connect an account." [Run an evaluation →](/builder/v2/testing-operations/running-an-evaluation) ## Improving a Chat AI Employee Group the failed test cases that share a cause, then fix each group once in the AI Employee's workflow and publish: - **Wrong or incomplete answer.** Change the Instructions of the agent that answered, or the knowledge base it answered from. - **Wrong tool, or a missing tool.** Change the step that calls the tool, or the Instructions that decide when it is called. - **An approval or form it did not need, or skipped one it did.** Change the step that raises it. See [Designing Human-in-the-Loop Forms](/builder/v2/builder-guides/designing-hitl-forms). Then open the run's ⋮ menu and select **Re-evaluate** to run the same dataset against the version you just published. To have Autopilot group the failures and propose the changes, select **Ask Autopilot** > **Improve** _AI Employee name_. It asks before changing anything. ## Troubleshooting Symptom Cause and fix No suggested goal under **Goal** The suggestion comes from the AI Employee's published summary, and it is shown only while that summary describes the published version. Write the goal yourself, or ask Autopilot to write it. A test case ends at "Asked to connect an account." The simulated user cannot connect an account, so the conversation stops at that request. Check whether the tool needs each user to sign in. If it does, leave requests that need it out of the dataset, or expect the request in the test case. A conversation ends with "Not answered before the conversation ended." The workflow asked for something the simulated user did not answer before the conversation reached its limit. Read the request on the **Conversation** tab. If the step should not ask at that point, change the workflow. The evaluation conversations are missing from the **Chat** tab Evaluation conversations are kept out of the **Chat** tab. Open the test case on the run page and select **Conversation**. **Response** passed but **Plan** failed The AI Employee gave the right answer by a route the test case did not expect, such as without a tool it should have used. Read **Problems found** on the **Plan** card of the **Verdict** tab. ## What's next - [Building a Test Dataset](/builder/v2/testing-operations/building-a-test-dataset): the goal, sources, generating, and reading and changing test cases. - [Running an Evaluation](/builder/v2/testing-operations/running-an-evaluation): starting a run, reading Response and Plan results, and improving the AI Employee. - [Evaluating Orchestrator AI Employees](/builder/v2/testing-operations/evaluating-orchestrators): how an evaluation differs for an orchestrator. - [AI Employee types](/builder/v2/getting-started/ai-employee-types): what a Chat AI Employee is and where it lives.