> Source: https://builder.ema.ai/v2/api-reference/evaluation-api
> Title: Evaluation API

# Evaluation API

Evaluation lets you measure how well an AI Employee — or a single agent inside it — performs against a dataset, using an LLM judge to score each output against a rubric. The Evaluation API is where you assemble the pieces: a dataset of inputs (and optional expected outputs), a rubric describing what good looks like, a config that ties them to a target, and a run that scores every row. The evaluation service is mounted at `/api/v1/eval`, so every path below is that prefix plus the path shown.

All requests require a [JWT or tenant API key](/builder/v2/api-reference/authentication). Examples use `https://your-tenant.ema.co` as the host. Throughout, `workflow_id` is the ID of the AI Employee being evaluated.

## The evaluation model

An evaluation has four resources, created in order:

1.  **Dataset** — a CSV of rows, each with inputs (and optionally an expected output or simulator instructions).
2.  **Rubric** — a JSON schema of score fields plus instructions telling the judge how to score.
3.  **Config** — ties a rubric to a target (a whole `workflow` or a single `agent` node), with the simulator and judge model settings.
4.  **Run** — executes the config against a dataset and produces one result per row, plus aggregate metrics.

## Datasets

A dataset starts as an uploaded CSV. You then map its columns to the inputs your target expects.

-   **Upload:** `POST /api/v1/eval/datasets` — a `multipart/form-data` body with `file`, `name`, and `workflow_id`. The response (`201`) reports the parsed CSV headers.
-   **Map columns:** `POST /api/v1/eval/datasets/{id}/mapping` — `input_map` (and optional `expected_output_map`) bind CSV columns to fields, plus the `csv_data` to materialize rows.
-   **List / get / delete:** `GET /api/v1/eval/datasets?workflow_id=...`, `GET` / `DELETE /api/v1/eval/datasets/{id}`.
-   **Rows:** `GET /api/v1/eval/datasets/{id}/rows` returns the paginated `DatasetRowResponse` items, each with its `inputs`, optional `expected_output`, and optional `simulator_instructions`.

```http
POST https://your-tenant.ema.co/api/v1/eval/datasets/4c8e.../mapping
Authorization: Bearer eyJhbGciOiJSUzI1Ni...
Content-Type: application/json

{
  "input_map": { "ticket_subject": "subject", "ticket_body": "body" },
  "expected_output_map": { "resolution": "expected_resolution" },
  "csv_data": "subject,body,expected_resolution\nReset password,...,..."
}
```

## Rubrics

A rubric defines the score fields the judge fills in and the instructions it follows.

-   **Create:** `POST /api/v1/eval/rubrics` — requires `workflow_id`, `name`, `rubric_schema` (the JSON schema of score fields), and `rubric_instructions`.
-   **Generate a draft:** `POST /api/v1/eval/rubrics/generate` turns a natural-language `description` into a rubric draft (not persisted — review and create it yourself).
-   **List / get / update / delete:** `GET /api/v1/eval/rubrics?workflow_id=...`, `GET` / `PUT` / `DELETE /api/v1/eval/rubrics/{id}`.

```http
POST https://your-tenant.ema.co/api/v1/eval/rubrics
Authorization: Bearer eyJhbGciOiJSUzI1Ni...
Content-Type: application/json

{
  "workflow_id": "3f7a...",
  "name": "Support answer quality",
  "rubric_instructions": "Score correctness and tone from 1-5.",
  "rubric_schema": {
    "type": "object",
    "properties": {
      "correctness": { "type": "number" },
      "tone": { "type": "number" }
    }
  }
}
```

Both create and generate accept an optional `variable_catalog` — the variables the rubric can reference, each typed as `dataset_input` or `agent_output`. Deleting a rubric that a config still references returns `409`.

## Configs

A config binds a rubric to a target and sets the run parameters.

-   **Create:** `POST /api/v1/eval/configs` — requires `workflow_id`, `name`, and `target_type` (`workflow` or `agent`).
-   **List / get / update / delete:** `GET /api/v1/eval/configs?workflow_id=...`, `GET` / `PUT` / `DELETE /api/v1/eval/configs/{id}`.

```http
POST https://your-tenant.ema.co/api/v1/eval/configs
Authorization: Bearer eyJhbGciOiJSUzI1Ni...
Content-Type: application/json

{
  "workflow_id": "3f7a...",
  "name": "Triage agent v4 vs golden set",
  "target_type": "agent",
  "target_node_id": "classify",
  "rubric_id": "7d1e...",
  "judge_model": "gpt-4o",
  "simulator_instructions": "Act as a frustrated customer.",
  "max_turns": 5
}
```

When `target_type` is `agent`, set `target_node_id` to the node you want to score in isolation. `simulator_model`, `simulator_instructions`, and `max_turns` drive a simulated conversation for multi-turn targets. `judge_model` names the LLM that scores each row. An optional `search_config` overrides search settings (such as `search_top_k` and `search_min_score`) for the duration of the run. Deleting a config that a run references returns `409`.

## Runs and results

A run executes a config against a dataset.

-   **Create:** `POST /api/v1/eval/runs` — requires `config_id` and `dataset_id`; set `is_preview: true` to score a small sample. The response is `202 Accepted` — the run scores rows asynchronously.
-   **List:** `GET /api/v1/eval/runs` — filter by `workflow_id`, `config_id`, `dataset_id`, `status`, or `is_preview`.
-   **Get:** `GET /api/v1/eval/runs/{id}` returns the `RunResponse` with `status`, row counts, and `aggregate_metrics`.
-   **Cancel:** `POST /api/v1/eval/runs/{id}/cancel` — already-terminal runs return `409`.
-   **Compare:** `GET /api/v1/eval/runs/compare?run_id_1=...&run_id_2=...` pairs results from two runs side by side. Comparing runs built on different datasets returns `422`.

```http
POST https://your-tenant.ema.co/api/v1/eval/runs
Authorization: Bearer eyJhbGciOiJSUzI1Ni...
Content-Type: application/json

{
  "config_id": "9b2c...",
  "dataset_id": "4c8e...",
  "is_preview": false
}
```

A run's `status` moves through `pending` → `running` → a terminal state (`completed`, `failed`, `cancelled`). When it completes, `aggregate_metrics` carries `total_rows`, `completed_rows`, `failed_rows`, and a `rubric_scores` map of per-field statistics (`mean`, `median`, `p5`, `p95`):

```json
{
  "id": "e5f6...",
  "config_id": "9b2c...",
  "dataset_id": "4c8e...",
  "status": "completed",
  "total_rows": 200,
  "completed_rows": 198,
  "failed_rows": 2,
  "aggregate_metrics": {
    "total_rows": 200,
    "completed_rows": 198,
    "failed_rows": 2,
    "rubric_scores": {
      "correctness": { "mean": 4.3, "median": 4, "p5": 3, "p95": 5 }
    }
  }
}
```

### Per-row results

-   **List a run's results:** `GET /api/v1/eval/runs/{id}/results` — paginated, with an optional `status` filter.
-   **Get one result:** `GET /api/v1/eval/results/{id}`.

Each `ResultResponse` carries the row's `status`, the `actual_output` the target produced, the `simulator_transcript` for multi-turn runs, `rubric_scores`, `token_usage`, `duration_ms`, and — for workflow targets — the `workflow_run_id` so you can pull the full [run trace](/builder/v2/api-reference/workflow-api).

## What's next

-   [Workflow API](/builder/v2/api-reference/workflow-api) — inspect the workflow runs an evaluation produces.
-   [AI Employee API](/builder/v2/api-reference/ai-employee-api) — version the AI Employee whose published version each run scores.