Evaluation API

Evaluation lets you measure how well an AI Employee — or a single agent inside it — performs against a dataset, using an LLM judge to score each output against a rubric. The Evaluation API is where you assemble the pieces: a dataset of inputs (and optional expected outputs), a rubric describing what good looks like, a config that ties them to a target, and a run that scores every row. The evaluation service is mounted at /api/v1/eval, so every path below is that prefix plus the path shown.

All requests require a JWT or tenant API key. Examples use https://your-tenant.ema.co as the host. Throughout, workflow_id is the ID of the AI Employee being evaluated.

The evaluation model

An evaluation has four resources, created in order:

Dataset — a CSV of rows, each with inputs (and optionally an expected output or simulator instructions).
Rubric — a JSON schema of score fields plus instructions telling the judge how to score.
Config — ties a rubric to a target (a whole workflow or a single agent node), with the simulator and judge model settings.
Run — executes the config against a dataset and produces one result per row, plus aggregate metrics.

Datasets

A dataset starts as an uploaded CSV. You then map its columns to the inputs your target expects.

Upload: POST /api/v1/eval/datasets — a multipart/form-data body with file, name, and workflow_id. The response (201) reports the parsed CSV headers.
Map columns: POST /api/v1/eval/datasets/{id}/mapping — input_map (and optional expected_output_map) bind CSV columns to fields, plus the csv_data to materialize rows.
List / get / delete: GET /api/v1/eval/datasets?workflow_id=..., GET / DELETE /api/v1/eval/datasets/{id}.
Rows: GET /api/v1/eval/datasets/{id}/rows returns the paginated DatasetRowResponse items, each with its inputs, optional expected_output, and optional simulator_instructions.

POST https://your-tenant.ema.co/api/v1/eval/datasets/4c8e.../mapping
Authorization: Bearer eyJhbGciOiJSUzI1Ni...
Content-Type: application/json

{
  "input_map": { "ticket_subject": "subject", "ticket_body": "body" },
  "expected_output_map": { "resolution": "expected_resolution" },
  "csv_data": "subject,body,expected_resolution\nReset password,...,..."
}

Rubrics

A rubric defines the score fields the judge fills in and the instructions it follows.

Create: POST /api/v1/eval/rubrics — requires workflow_id, name, rubric_schema (the JSON schema of score fields), and rubric_instructions.
Generate a draft: POST /api/v1/eval/rubrics/generate turns a natural-language description into a rubric draft (not persisted — review and create it yourself).
List / get / update / delete: GET /api/v1/eval/rubrics?workflow_id=..., GET / PUT / DELETE /api/v1/eval/rubrics/{id}.

POST https://your-tenant.ema.co/api/v1/eval/rubrics
Authorization: Bearer eyJhbGciOiJSUzI1Ni...
Content-Type: application/json

{
  "workflow_id": "3f7a...",
  "name": "Support answer quality",
  "rubric_instructions": "Score correctness and tone from 1-5.",
  "rubric_schema": {
    "type": "object",
    "properties": {
      "correctness": { "type": "number" },
      "tone": { "type": "number" }
    }
  }
}

Both create and generate accept an optional variable_catalog — the variables the rubric can reference, each typed as dataset_input or agent_output. Deleting a rubric that a config still references returns 409.

Configs

A config binds a rubric to a target and sets the run parameters.

Create: POST /api/v1/eval/configs — requires workflow_id, name, and target_type (workflow or agent).
List / get / update / delete: GET /api/v1/eval/configs?workflow_id=..., GET / PUT / DELETE /api/v1/eval/configs/{id}.

POST https://your-tenant.ema.co/api/v1/eval/configs
Authorization: Bearer eyJhbGciOiJSUzI1Ni...
Content-Type: application/json

{
  "workflow_id": "3f7a...",
  "name": "Triage agent v4 vs golden set",
  "target_type": "agent",
  "target_node_id": "classify",
  "rubric_id": "7d1e...",
  "judge_model": "gpt-4o",
  "simulator_instructions": "Act as a frustrated customer.",
  "max_turns": 5
}

When target_type is agent, set target_node_id to the node you want to score in isolation. simulator_model, simulator_instructions, and max_turns drive a simulated conversation for multi-turn targets. judge_model names the LLM that scores each row. An optional search_config overrides search settings (such as search_top_k and search_min_score) for the duration of the run. Deleting a config that a run references returns 409.

Runs and results

A run executes a config against a dataset.

Create: POST /api/v1/eval/runs — requires config_id and dataset_id; set is_preview: true to score a small sample. The response is 202 Accepted — the run scores rows asynchronously.
List: GET /api/v1/eval/runs — filter by workflow_id, config_id, dataset_id, status, or is_preview.
Get: GET /api/v1/eval/runs/{id} returns the RunResponse with status, row counts, and aggregate_metrics.
Cancel: POST /api/v1/eval/runs/{id}/cancel — already-terminal runs return 409.
Compare: GET /api/v1/eval/runs/compare?run_id_1=...&run_id_2=... pairs results from two runs side by side. Comparing runs built on different datasets returns 422.

POST https://your-tenant.ema.co/api/v1/eval/runs
Authorization: Bearer eyJhbGciOiJSUzI1Ni...
Content-Type: application/json

{
  "config_id": "9b2c...",
  "dataset_id": "4c8e...",
  "is_preview": false
}

A run's status moves through pending → running → a terminal state (completed, failed, cancelled). When it completes, aggregate_metrics carries total_rows, completed_rows, failed_rows, and a rubric_scores map of per-field statistics (mean, median, p5, p95):

{
  "id": "e5f6...",
  "config_id": "9b2c...",
  "dataset_id": "4c8e...",
  "status": "completed",
  "total_rows": 200,
  "completed_rows": 198,
  "failed_rows": 2,
  "aggregate_metrics": {
    "total_rows": 200,
    "completed_rows": 198,
    "failed_rows": 2,
    "rubric_scores": {
      "correctness": { "mean": 4.3, "median": 4, "p5": 3, "p95": 5 }
    }
  }
}

Per-row results

List a run's results: GET /api/v1/eval/runs/{id}/results — paginated, with an optional status filter.
Get one result: GET /api/v1/eval/results/{id}.

Each ResultResponse carries the row's status, the actual_output the target produced, the simulator_transcript for multi-turn runs, rubric_scores, token_usage, duration_ms, and — for workflow targets — the workflow_run_id so you can pull the full run trace.

What's next

Workflow API — inspect the workflow runs an evaluation produces.
AI Employee API — version the AI Employee whose published version each run scores.