Launch Checklist

This is the readiness checklist for taking an AI Employee (AIE) live. Each item maps to a real platform feature, with a link to the page that explains it in depth. Work through it before you publish — launching is the end of the testing path, not the start.

Ema is a managed cloud platform: there is no deploy step and no infrastructure to provision. "Going live" means publishing a workflow version so it becomes the version users hit. The checklist below is about confirming the version you are about to publish is ready, and that you can recover if it isn't.

1. Test coverage

Confirm the AIE actually works before measuring how well.

• Every branch has been exercised. Run the workflow interactively and cover each path — not just the happy path. For a chat AIE, test follow-up handling (does it retain context across turns?) and context isolation (does an unrelated follow-up leak the previous topic?).
• Inputs and edge cases are covered. Run with the required inputs, with optional inputs omitted, and with malformed or unexpected input to see how the AIE degrades.
• You tested with test mode where appropriate. Test mode mocks integration responses from each Tool's output schema, so you can exercise the full graph without making real external calls. Use it while iterating, then test against real systems before launch (see section 4).
• You read the debug logs for a representative run. Confirm the nodes ran in the order you expect, each agent received the inputs you intended, and the LLM prompts and Tool calls look right. A run that produced the correct answer for the wrong reason is a latent bug.

2. Evaluation

Replace "it seems better" with a number.

• You ran an evaluation over a representative dataset. Build a dataset of real or representative inputs, define a rubric, and run the AIE over every row to get per-row results and aggregate scores.
• The scores meet your bar — including the tail. Look at the 5th-percentile (p5) score, not just the mean. A high mean with a low p5 means the AIE is usually good but occasionally bad; open the worst rows via their linked workflow runs and decide whether the tail is acceptable.
• You compared the candidate against a baseline. Compare two runs of the same dataset side by side — your current live version (or a previous candidate) versus the version you intend to launch — to confirm the change is a net improvement, row by row.
• For human-in-the-loop workflows, the HITL simulator ran clean. Confirm the simulator's approvals, form fills, and answers reflect realistic user behavior, and that rows completed within the configured turn limit rather than timing out.

3. Permissions and access

Confirm the right people — and only the right people — can reach the AIE.

• The audience is set correctly. Confirm which users and admins have access to the AIE. Access is governed by roles; see Administration for the role model.
• Workflow access reflects your intent. Review the workflow's access configuration and assignments. Changes here are auditable — workflow_access.updated, workflow_permission.created, and workflow_assignment.created all appear in the audit trail, so you can confirm exactly who was granted access and by whom.
• API keys are scoped and accounted for. If the AIE is reached through an API key, confirm the key exists, is owned by the right party, and that no stale keys remain. api_key.created and api_key.revoked events are in the audit trail.
• Roles for reviewers are in place. Conversation review and the tenant audit log are admin-gated (system_admin / builder_admin). Confirm the people who will monitor the AIE after launch hold the role that lets them see conversations and the audit log.

4. Knowledge base readiness

If the AIE answers from a knowledge base, confirm the knowledge is current and the retrieval is tuned.

• Content is complete and current. Confirm every source the AIE should know about has been ingested, and that stale documents have been removed. Ingestion and removal are auditable (document.uploaded, document.deleted).
• Retrieval is tuned. If you adjusted search parameters (for example top-k or a minimum score threshold) while building, confirm the live settings are the ones you tuned. You can evaluate retrieval-tuning changes in an eval config via search_config overrides before committing them to the live AIE.
• The knowledge-base version is the one you intend to ship. Publishing snapshots the knowledge state with the version; confirm you are publishing on top of the right knowledge base, and that any kb_snapshot.reverted you performed left it where you want.

5. Publish validation

Publishing runs validation first and fails the publish (rather than shipping something broken) if the workflow can't go live.

• The workflow passes publish validation. Publishing with POST /workflows/{id}/publish validates the workflow and returns a validation error (422) instead of going live if it can't. Validation covers, among others:
  • Structural DAG errors (disconnected or malformed nodes).
  • LLM-provider and model checks.
  • Human-in-the-loop assignee checks.
  • MCP server and per-tool configuration errors.
  • Agent-type version pins that are deprecated, deleted, broken, unknown, or pre-release.
• No test artifacts remain. Remove debugging nodes, temporary outputs, or scratch configuration you added while building.
• Integrations point at production. If you tested against sandbox instances or with test mode, switch your Tools to the real production systems before users rely on them.
• Name and description are user-ready. They appear to the people who use the AIE.

See Launching and monitoring for the full publish flow.

6. Monitoring

Make sure you'll see how the AIE performs the moment it's live.

• You know where the metrics live. Every AIE gets a Metrics tab in the builder with no extra configuration. Chat AIEs get a conversational analytics dashboard (usage, sessions, messages, active users, feedback trends, tool-call activity); other trigger types get the metrics appropriate to how they run. See Launching and monitoring.
• Reviewers can read conversations. Confirm an admin can open the conversation-review surface, read transcripts, see user feedback, and set resolution status. This is your qualitative signal after launch.
• Export is wired up if you need it. If you feed conversational data into your own BI stack, confirm you can export row-level chat metrics (CSV, NDJSON, or JSON).
• The audit trail is the record of changes. Know that every post-launch change to the AIE is captured in the audit trail — useful when a metric moves and you need to correlate it with a change.

7. Rollback plan

Confirm you can undo the launch quickly.

• You can revert to a previous version. Each publish creates a numbered version. List past versions with GET /workflows/{id}/versions and revert to a previously published one if a launch needs to be rolled back. Reverts are auditable (workflow.restored, workflow.version.reverted).
• You rolled out in phases. Limit the blast radius of anything you missed: launch to a small, high-impact group first, define exit criteria for widening access, and give users a clear channel to report problems. See Roll out in phases.
• You know your trigger for rollback. Decide in advance what signal (a drop in feedback rate, a spike in run.failed events, a cluster of Needs improvement conversations) means "revert now" rather than "investigate later."

After launch

Launch is the start of a loop, not the end of one:

1. Watch the metrics dashboard for drops in feedback rate, spikes in errors, or unexpected tool-call patterns.
2. Open the affected conversations or debug logs to see what went wrong, and check the audit trail for any change that lines up with the shift.
3. Reproduce and fix in the draft, test the fix, and evaluate it against your dataset.
4. Publish the new version and keep watching.

What's next

• Launching and monitoring — the publish flow and the metrics surfaces.
• Auditing AI Employees — the audit trail and conversation review the checklist relies on.
• Evaluating AI Employees — score the candidate version before you publish it.