> Source: https://builder.ema.ai/v2/testing-operations/evaluation > Title: Evaluating AI Employees # Evaluating AI Employees Evaluation measures how an AI Employee (AIE) performs across a whole dataset of inputs, instead of one run at a time. You upload a set of test cases, define how each response should be scored, run the AIE over every row, and get per-row results plus aggregate statistics. Because the same dataset and scoring config can be re-run after any change, evaluation turns "it seems better" into a repeatable, comparable number. Evaluation is built on a dedicated eval service. It runs each dataset row through the AIE's workflow, uses an LLM **judge** to score the output against a **rubric**, and — for workflows that pause for a human — uses an LLM **simulator** to play the human role so the run can complete unattended. Scoring is performed entirely by the rubric-based LLM judge. This page covers the full model: datasets, configs, rubrics, the simulator, preview runs, and comparison. ## The building blocks Evaluation is composed from four reusable entities, all scoped to one workflow (one AI Employee): Entity What it is **Dataset** A set of test cases, uploaded from a CSV. Each row holds the workflow inputs, optional expected output values, and optional per-row simulator instructions. **Rubric** A reusable scoring definition: a JSON schema describing the score fields the judge must return, plus natural-language instructions for how to score. **Eval config** Ties a target (the whole workflow, or a single agent node) to a rubric, simulator settings, and an LLM judge model. The "how to run and score" half of an evaluation. **Run** One execution of a config against a dataset. Produces a result per row and aggregate metrics. Decoupling these means the same dataset can be scored by different configs (different rubrics, different judge models), and the same config can be run against different datasets — which is what makes A/B comparison possible. ## Datasets A dataset is built from a CSV file. The columns can be anything; you map them to roles after upload. ### Upload and map columns 1. **Upload the CSV.** `POST /datasets` (multipart form with `file`, `name`, `workflow_id`). The service parses the header row and returns the detected columns. Column names are normalized to stable IDs so they can be referenced later regardless of capitalization or spacing. 2. **Map columns to roles.** `POST /datasets/{id}/mapping` with: - `input_map` — maps each workflow input parameter to a CSV column. **Required** inputs that are missing from the CSV cause the mapping to fail with a validation error. - `expected_output_map` (optional) — maps expected-output fields to CSV columns, for reference and CSV export. - `simulator_instructions_col` (optional) — names a CSV column whose value becomes per-row instructions for the HITL simulator. Every cell is also stored under its normalized column ID (as `raw_columns`), so a rubric can reference any column in the CSV, not just the ones mapped to workflow inputs. 3. The mapping creates the dataset's rows. List them with `GET /datasets/{id}/rows`. ```json POST /datasets/{dataset_id}/mapping { "input_map": { "query": "user_question" }, "expected_output_map": { "answer": "gold_answer" }, "simulator_instructions_col": "persona_notes", "csv_data": "user_question,gold_answer,persona_notes\n..." } ``` A dataset row carries: `inputs` (the mapped workflow inputs), optional `expected_output`, optional `simulator_instructions`, and `raw_columns` (every cell, by normalized column ID). ## Rubrics — how the LLM judge scores A rubric is the scoring contract for the LLM judge, and the only thing that produces a score in an eval run. It has two parts: - **`rubric_schema`** — a JSON schema describing the object the judge must return. This is what defines the score fields. Only fields named in the schema are kept in the aggregate metrics, so a malformed judge response can't invent spurious score fields. - **`rubric_instructions`** — the natural-language prompt telling the judge how to score, what to look for, and what each value on the scale means. ### Variables in rubric instructions Rubric instructions can include `{{...}}` template variables drawn from the rubric's **variable catalog**. Two kinds are supported: - `{{dataset.}}` — a value from the dataset row (any normalized CSV column). Resolved per row before the prompt goes to the judge. A reference to a column that isn't in the CSV is an authoring error and fails the render. - `{{agent.}}` — the output of a specific agent node in the workflow, captured during the run. Because workflows branch, an agent on an unexecuted branch produces no output; the reference is replaced with a placeholder and the judge is told to score the branch that did run, rather than penalize a missing one. This lets a rubric compare the AIE's answer against the row's gold answer, or score a specific intermediate agent's output, in plain language. ### Create and manage rubrics Action Endpoint Create `POST /rubrics` Generate a draft from a description `POST /rubrics/generate` List (per workflow) `GET /rubrics?workflow_id=...` Get / update / delete `GET` / `PUT` / `DELETE /rubrics/{id}` `POST /rubrics/generate` produces a rubric draft from a natural-language description — a fast way to get a starting schema and instructions you then refine. The draft is returned, not persisted, so you review it before saving. Deleting a rubric that a config still references returns a conflict (`409`); detach it from the config first. ## Eval configs A config is the "how to run and score" definition. Create it with `POST /configs`: Field Meaning `target_type` `workflow` (score the whole AIE's final output) or `agent` (score one node's output). `target_node_id` Required when `target_type` is `agent` — the DAG node to score. `workflow_version_id` Optional — pin the run to a specific published version so results are reproducible. `rubric_id` The rubric the judge scores against. Optional — without a rubric the run still executes and captures outputs, just with no scores. `judge_model` The LLM used as the judge. `simulator_instructions` Default instructions for the HITL simulator (applies to every row; per-row instructions from the dataset are layered on top). `simulator_model` The LLM used as the simulator. `max_turns` Maximum HITL interaction turns per row before the row fails (defaults to 10 if unset). `search_config` Optional search-parameter overrides merged into the agent's config for the run (for example `search_top_k`, `search_min_score`) so you can evaluate retrieval-tuning changes without editing the live AIE. Manage configs with `GET /configs?workflow_id=...`, and `GET` / `PUT` / `DELETE /configs/{id}`. Deleting a config referenced by a run returns a conflict (`409`). > [INFO] > **A note on assertions.** An eval config also accepts an optional `assertions` field — deterministic rules (a field and an operator) intended to pair with per-row expected values in the dataset. These rules are stored on the config, but **no shipped code evaluates them.** The eval run executor scores every row with the LLM judge against the rubric only; it does not compute deterministic assertion pass/fail. The result fields `assertion_results` and `assertions_passed` exist on the API and database but are not populated by a run, and the run wizard and result views do not surface them. Treat assertions as a stored, not-yet-evaluated capability — rely on rubric scores for grading. ## The LLM simulator Many workflows pause for a human — to approve an action, fill a form, or answer a question (human-in-the-loop, or HITL). To evaluate these unattended across hundreds of rows, the eval service uses an LLM **simulator** that plays the human: - **Approval** requests get `{"approved": true|false}`. - **Form** requests get a JSON object matching the form's schema. - **Ask-a-human** requests get `{"message": "..."}`. The simulator decides based on the config's `simulator_instructions` (general guidance) plus the row's `simulator_instructions` (case-specific guidance). Each HITL exchange is recorded in the result's simulator transcript, and the transcript is also given to the judge so scoring accounts for the full interaction. The poll-and-simulate loop runs up to `max_turns` times per row; a row that never completes within that limit is marked failed. ## Running an evaluation The AI Employee builder walks you through a four-step **New run** wizard: **Dataset → Config → Preview → Launch**. 1. **Dataset.** Pick an existing dataset or upload a new one. 2. **Config.** Pick an existing config or create one inline. 3. **Preview.** The wizard launches a small preview run automatically and shows the results when it finishes. 4. **Launch.** Review the dataset, config, and row count, then launch the full run. ### Preview runs A preview run executes the same pipeline as a full run but limits the dataset to the **first 5 rows** (fewer if the dataset is smaller). It's a fast, cheap check that the config, rubric, and simulator are wired correctly before you spend a full run on hundreds of rows. Preview runs are marked with `is_preview: true`, so you can include or exclude them from a run list with the `is_preview` filter (`GET /runs?is_preview=false` returns only full runs). ### Run lifecycle and the API Action Endpoint Create a run `POST /runs` (`config_id`, `dataset_id`, optional `is_preview`) — returns `202`, runs asynchronously List runs `GET /runs?workflow_id=...` (filter by `config_id`, `dataset_id`, `status`, `is_preview`) Get a run `GET /runs/{id}` — includes aggregate metrics Cancel a run `POST /runs/{id}/cancel` — `409` if already terminal List a run's results `GET /runs/{id}/results` (filter by `status`) Get one result `GET /results/{id}` Compare two runs `GET /runs/compare?run_id_1=...&run_id_2=...` A run moves through `pending` → `running` → `completed`, or ends in `failed` or `cancelled`. Rows are processed in parallel; progress is tracked as `completed_rows` and `failed_rows` against `total_rows`. A run whose every row failed is itself marked `failed`. Each row triggers the workflow, polls until it reaches a terminal state (answering any HITL pauses via the simulator), extracts the output, and — when the config has a rubric — sends that output and any simulator transcript to the LLM judge for scoring. If the judge call fails, the row still completes and stores its output; it just carries no rubric scores. ```json POST /runs { "config_id": "c1a2...", "dataset_id": "d3b4...", "is_preview": false } ``` ## Reading results ### Per-row results Each result row records: - `status` — `pending`, `running`, `completed`, or `failed`. - `actual_output` — the workflow's (or target agent's) output for the row. - `rubric_scores` — the judge's scores, keyed by the rubric schema's fields. The only score an eval run produces. - `simulator_transcript` — the HITL exchanges, if any. - `workflow_run_id` — a link to the underlying workflow run, so you can open its [debug logs](/builder/v2/testing-operations/debug-logs) to see exactly what happened. - `duration_ms`, `token_usage`, and `error_message` (on failure). The result schema also defines `assertion_results` and `assertions_passed`, but the run executor does not populate them (see the note under [Eval configs](#eval-configs)). Grade rows by their `rubric_scores`. ### Aggregate metrics When a run finishes, the service computes summary statistics. For each **numeric** rubric score field it reports **mean, median, 5th percentile (p5), and 95th percentile (p95)** across the completed rows. Non-numeric fields (for example a `*_reasoning` text field) are skipped. The run also reports `total_rows`, `completed_rows`, and `failed_rows`. ```json { "rubric_scores": { "relevance": { "mean": 4.2, "median": 4, "p5": 3, "p95": 5 }, "accuracy": { "mean": 3.8, "median": 4, "p5": 2, "p95": 5 } }, "total_rows": 120, "completed_rows": 118, "failed_rows": 2 } ``` The percentiles matter as much as the mean: a high mean with a low p5 tells you the AIE is usually good but occasionally bad — exactly the tail you want to inspect via the linked workflow runs. ### Export Run results can be downloaded as a CSV from the run detail page; the AI Employee builder assembles the file in the browser from the run's dataset rows and results. The export joins each dataset row with its result and produces columns for each input (`input_*`), each expected-output field (`expected_*`), `status`, `actual_output`, each rubric score (`rubric_*`), `duration_ms`, and `error_message`. ## Comparing runs To see whether a change helped, compare two runs **of the same dataset** side by side: `GET /runs/compare?run_id_1=...&run_id_2=...`. The service pairs results by dataset row so you can read the two outputs and their rubric scores together, row by row. The two runs must use the same dataset; comparing runs on different datasets returns a validation error (`422`). This is the core A/B workflow — run a baseline config and a candidate config against one dataset, then compare. ## What's next - [Testing AI Employees](/builder/v2/testing-operations/testing-ai-employees) — interactive, single runs while you build. - [Debug logs](/builder/v2/testing-operations/debug-logs) — open the workflow run behind any result row to see the full trace. - [Launching and monitoring](/builder/v2/testing-operations/launching-and-monitoring) — publish once the scores meet your bar.