Testing & Operations

This section covers the operational lifecycle of an AI Employee (AIE): testing it as you build, evaluating it at scale against a dataset, launching it to real users, and monitoring it in production. Every workflow goes through the same path — run it interactively, debug what it did, score it against expected behavior, publish it, then watch the metrics.

Everything here works against your AI Employee's underlying workflow. An AI Employee is built as a workflow — a directed graph of typed agent nodes — so "testing the AI Employee" and "running the workflow" are the same action. See Workflows for the model behind it.

In this section

• Testing AI Employees — run an AIE interactively, use test mode (a dry run that mocks integration calls), and inspect individual executions before they touch production.
• Evaluating AI Employees — measure quality at scale with datasets, eval configs, deterministic assertions, LLM rubric scoring, the HITL simulator, preview runs, and side-by-side run comparison.
• Launching and monitoring — publish a version, roll out in phases, and track adoption with out-of-the-box metrics.
• Launch checklist — a pre-launch readiness checklist mapping every gate (test coverage, evaluation, access, knowledge base, monitoring, rollback) to a real platform feature.
• Auditing AI Employees — review who changed an AIE and what it did: the immutable platform audit trail, per-resource activity, and conversation review.
• Debug logs — read a run's step trace: per-node status, inputs and their sources, work logs, LLM prompts and responses, tool calls, and errors.
• Chat metrics — export row-level conversational data (messages and conversations) as CSV, NDJSON, or JSON for your own BI stack.

How the pieces fit together

A typical path from build to production:

1. Build and run. As you build the workflow, run it interactively from the AI Employee builder. Use test mode for runs that should not make real external calls or count toward production metrics. See Testing AI Employees.
2. Debug. When a run does something unexpected, open its debug logs to see exactly which nodes ran, what each agent received, and what the LLM was asked and answered.
3. Evaluate. Before a meaningful change goes live, run an evaluation over a dataset of representative inputs to get objective, repeatable scores — and compare the new run against a baseline.
4. Launch. Work through the launch checklist, then publish the version and roll it out to a controlled group of users first.
5. Monitor. Watch the in-app metrics, review real conversations, and export chat metrics into your warehouse for deeper analysis. Use the audit trail to confirm who changed what. Feed what you learn back into step 1.