Evaluating AI Employees

Evaluation measures how an AI Employee (AIE) performs across a whole dataset of inputs, instead of one run at a time. You upload a set of test cases, define how each response should be scored, run the AIE over every row, and get per-row results plus aggregate statistics. Because the same dataset and scoring config can be re-run after any change, evaluation turns "it seems better" into a repeatable, comparable number.

Evaluation is built on a dedicated eval service. It runs each dataset row through the AIE's workflow, uses an LLM judge to score the output against a rubric, and — for workflows that pause for a human — uses an LLM simulator to play the human role so the run can complete unattended. Scoring is performed entirely by the rubric-based LLM judge. This page covers the full model: datasets, configs, rubrics, the simulator, preview runs, and comparison.

The building blocks

Evaluation is composed from four reusable entities, all scoped to one workflow (one AI Employee):

Decoupling these means the same dataset can be scored by different configs (different rubrics, different judge models), and the same config can be run against different datasets — which is what makes A/B comparison possible.

Datasets

A dataset is built from a CSV file. The columns can be anything; you map them to roles after upload.

Upload and map columns

1. Upload the CSV. POST /datasets (multipart form with file, name, workflow_id). The service parses the header row and returns the detected columns. Column names are normalized to stable IDs so they can be referenced later regardless of capitalization or spacing.
2. Map columns to roles. POST /datasets/{id}/mapping with:
   • input_map — maps each workflow input parameter to a CSV column. Required inputs that are missing from the CSV cause the mapping to fail with a validation error.
   • expected_output_map (optional) — maps expected-output fields to CSV columns, for reference and CSV export.
   • simulator_instructions_col (optional) — names a CSV column whose value becomes per-row instructions for the HITL simulator.
   Every cell is also stored under its normalized column ID (as raw_columns), so a rubric can reference any column in the CSV, not just the ones mapped to workflow inputs.
3. The mapping creates the dataset's rows. List them with GET /datasets/{id}/rows.

POST /datasets/{dataset_id}/mapping
{
  "input_map":            { "query": "user_question" },
  "expected_output_map":  { "answer": "gold_answer" },
  "simulator_instructions_col": "persona_notes",
  "csv_data": "user_question,gold_answer,persona_notes\n..."
}

A dataset row carries: inputs (the mapped workflow inputs), optional expected_output, optional simulator_instructions, and raw_columns (every cell, by normalized column ID).

Rubrics — how the LLM judge scores

A rubric is the scoring contract for the LLM judge, and the only thing that produces a score in an eval run. It has two parts:

• rubric_schema — a JSON schema describing the object the judge must return. This is what defines the score fields. Only fields named in the schema are kept in the aggregate metrics, so a malformed judge response can't invent spurious score fields.
• rubric_instructions — the natural-language prompt telling the judge how to score, what to look for, and what each value on the scale means.

Variables in rubric instructions

Rubric instructions can include {{...}} template variables drawn from the rubric's variable catalog. Two kinds are supported:

• {{dataset.<column>}} — a value from the dataset row (any normalized CSV column). Resolved per row before the prompt goes to the judge. A reference to a column that isn't in the CSV is an authoring error and fails the render.
• {{agent.<node_id>}} — the output of a specific agent node in the workflow, captured during the run. Because workflows branch, an agent on an unexecuted branch produces no output; the reference is replaced with a placeholder and the judge is told to score the branch that did run, rather than penalize a missing one.

This lets a rubric compare the AIE's answer against the row's gold answer, or score a specific intermediate agent's output, in plain language.

Create and manage rubrics

POST /rubrics/generate produces a rubric draft from a natural-language description — a fast way to get a starting schema and instructions you then refine. The draft is returned, not persisted, so you review it before saving. Deleting a rubric that a config still references returns a conflict (409); detach it from the config first.

Eval configs

A config is the "how to run and score" definition. Create it with POST /configs:

Manage configs with GET /configs?workflow_id=..., and GET / PUT / DELETE /configs/{id}. Deleting a config referenced by a run returns a conflict (409).

The LLM simulator

Many workflows pause for a human — to approve an action, fill a form, or answer a question (human-in-the-loop, or HITL). To evaluate these unattended across hundreds of rows, the eval service uses an LLM simulator that plays the human:

• Approval requests get {"approved": true|false}.
• Form requests get a JSON object matching the form's schema.
• Ask-a-human requests get {"message": "..."}.

The simulator decides based on the config's simulator_instructions (general guidance) plus the row's simulator_instructions (case-specific guidance). Each HITL exchange is recorded in the result's simulator transcript, and the transcript is also given to the judge so scoring accounts for the full interaction. The poll-and-simulate loop runs up to max_turns times per row; a row that never completes within that limit is marked failed.

Running an evaluation

The AI Employee builder walks you through a four-step New run wizard: Dataset → Config → Preview → Launch.

1. Dataset. Pick an existing dataset or upload a new one.
2. Config. Pick an existing config or create one inline.
3. Preview. The wizard launches a small preview run automatically and shows the results when it finishes.
4. Launch. Review the dataset, config, and row count, then launch the full run.

Preview runs

A preview run executes the same pipeline as a full run but limits the dataset to the first 5 rows (fewer if the dataset is smaller). It's a fast, cheap check that the config, rubric, and simulator are wired correctly before you spend a full run on hundreds of rows. Preview runs are marked with is_preview: true, so you can include or exclude them from a run list with the is_preview filter (GET /runs?is_preview=false returns only full runs).

Run lifecycle and the API

A run moves through pending → running → completed, or ends in failed or cancelled. Rows are processed in parallel; progress is tracked as completed_rows and failed_rows against total_rows. A run whose every row failed is itself marked failed.

Each row triggers the workflow, polls until it reaches a terminal state (answering any HITL pauses via the simulator), extracts the output, and — when the config has a rubric — sends that output and any simulator transcript to the LLM judge for scoring. If the judge call fails, the row still completes and stores its output; it just carries no rubric scores.

POST /runs
{
  "config_id":  "c1a2...",
  "dataset_id": "d3b4...",
  "is_preview": false
}

Reading results

Per-row results

Each result row records:

• status — pending, running, completed, or failed.
• actual_output — the workflow's (or target agent's) output for the row.
• rubric_scores — the judge's scores, keyed by the rubric schema's fields. The only score an eval run produces.
• simulator_transcript — the HITL exchanges, if any.
• workflow_run_id — a link to the underlying workflow run, so you can open its debug logs to see exactly what happened.
• duration_ms, token_usage, and error_message (on failure).

The result schema also defines assertion_results and assertions_passed, but the run executor does not populate them (see the note under Eval configs). Grade rows by their rubric_scores.

Aggregate metrics

When a run finishes, the service computes summary statistics. For each numeric rubric score field it reports mean, median, 5th percentile (p5), and 95th percentile (p95) across the completed rows. Non-numeric fields (for example a *_reasoning text field) are skipped. The run also reports total_rows, completed_rows, and failed_rows.

{
  "rubric_scores": {
    "relevance": { "mean": 4.2, "median": 4, "p5": 3, "p95": 5 },
    "accuracy":  { "mean": 3.8, "median": 4, "p5": 2, "p95": 5 }
  },
  "total_rows": 120,
  "completed_rows": 118,
  "failed_rows": 2
}

The percentiles matter as much as the mean: a high mean with a low p5 tells you the AIE is usually good but occasionally bad — exactly the tail you want to inspect via the linked workflow runs.

Export

Run results can be downloaded as a CSV from the run detail page; the AI Employee builder assembles the file in the browser from the run's dataset rows and results. The export joins each dataset row with its result and produces columns for each input (input_*), each expected-output field (expected_*), status, actual_output, each rubric score (rubric_*), duration_ms, and error_message.

Comparing runs

To see whether a change helped, compare two runs of the same dataset side by side: GET /runs/compare?run_id_1=...&run_id_2=.... The service pairs results by dataset row so you can read the two outputs and their rubric scores together, row by row. The two runs must use the same dataset; comparing runs on different datasets returns a validation error (422). This is the core A/B workflow — run a baseline config and a candidate config against one dataset, then compare.

What's next

• Testing AI Employees — interactive, single runs while you build.
• Debug logs — open the workflow run behind any result row to see the full trace.
• Launching and monitoring — publish once the scores meet your bar.