Auditing AI Employees
Auditing answers two questions about an AI Employee (AIE): who changed it and what it did. Ema records both in two complementary places. The platform audit trail is an immutable, who-did-what log of every administrative and builder action across the tenant — who published a workflow version, who connected an integration, who changed a user's role. Conversation review is the qualitative companion: the full transcript of an individual conversation a chat AIE had, with reviewer notes and a resolution status.
This page covers both. Use the audit trail to investigate changes and satisfy compliance; use conversation review to read what the AIE actually said to a user and grade it.
The platform audit trail
Every service in Ema emits an audit event after it mutates data. The events flow into a dedicated, append-only audit log. There is no API to edit or delete an event — the log is immutable by design, and old events are removed only by a background retention process (see Retention).
What an event records
Each audit event captures who acted, what they did, and what they acted on:
| Field | Meaning |
|---|---|
actor_id, actor_role | The user who performed the action and the role they held at the time. Background jobs with no user context record a system actor with role system. |
actor_email, actor_name | The actor's email and display name captured at event time, so a row stays readable even if the user is later renamed or removed. |
action | What happened, as a dotted identifier — for example workflow.published, user.role_changed, connection.oauth_connected. |
resource_type, resource_id | The kind of thing acted on (workflow, user, knowledge_base, run, api_key, …) and its ID. |
resource_name | A human-readable name for the resource, captured at event time. It can become stale if the resource is later renamed — that is acceptable for an audit record, which describes the world as it was. |
resource_version_id | The version of the resource, when the resource is versioned (for example which published workflow version a workflow.published event refers to). |
metadata | Free-form, action-specific context — for example the uploaded filename on a document.uploaded event, or the tenant on a session.login. |
ip_address | The actor's IP address, when available. |
created_at | When the action happened. |
Actions follow a resource.verb convention. Real examples you will see for an AIE and its surroundings:
| Area | Example actions |
|---|---|
| Workflow (the AIE itself) | workflow.created, workflow.updated, workflow.published, workflow.deleted, workflow.restored, workflow_config.updated, workflow_access.updated |
| Versions | workflow.version.promoted, workflow.version.reverted, workflow.version.renamed, workflow.version.deleted, kb_snapshot.reverted |
| Runs | run.started, run.completed, run.failed, run.cancelled |
| Knowledge base | knowledge_base.created, document.uploaded, document.deleted |
| Integrations & tools | connection.oauth_connected, integration.installed |
| People & access | user.created, user.role_changed, user.deactivated, api_key.created, api_key.revoked, session.login, session.logout |
Audit is fire-and-forget and best-effort by design. Services publish events asynchronously, so emitting an event never blocks or fails the action that triggered it. The transport is durable — if the audit log is briefly unavailable, events queue and are recorded on recovery rather than lost. The audit trail is a record of actions, not a guarantee that an action succeeded; pair it with debug logs when you need the execution detail behind a run.
Where to read it
There are two views onto the same audit log.
Tenant-wide audit log. Administrators get a full audit log under Admin → Governance → Audit. It lists every event for the tenant, newest first, with filters for actor (user), resource type, and action, and a free-text-friendly set of preset action and resource filters. Each row renders as a plain-language sentence ("Jane Doe published workflow Support Triage") and expands to show the structured metadata behind it. The log paginates with a Load more control and supports Export CSV of the rows currently loaded — columns are time, actor name, actor email, actor role, action, resource type, resource name, and resource ID.
Per-resource activity. Inside the AI Employee builder, the workflow's Activity rail shows the same audit events filtered to that one workflow — a focused change history for the AIE you are looking at, without leaving the builder. It is the fastest way to answer "what changed on this AIE, and who changed it?" while you work.
Who can see what
The audit trail is filtered by role at the query level, so a user only ever receives the rows their role is allowed to see — pagination and export stay correct because filtering happens before rows are fetched, not after.
| Role | Audit visibility |
|---|---|
system_admin, user_admin, env_admin | Full visibility — every event in the tenant. |
builder_admin, builder | Builder-scoped resource types only: workflow, agent, knowledge_base, run, document, integration. |
user | Only events where they were the actor (their own actions). |
no_access | Nothing. |
Filtering and the query API
The audit log is also available programmatically. GET /events returns the tenant's events, newest first, with the same RBAC filtering applied to the calling identity. Combine these query parameters to narrow the feed:
| Parameter | Filters by |
|---|---|
actor_id | A single user. |
resource_type | One or more resource types (comma-separated), e.g. workflow,workflow_run. |
resource_id | A single resource — the "what happened to this AIE" query. |
action | One or more actions (comma-separated), e.g. workflow.published,workflow.deleted. |
from, to | An inclusive time range (RFC 3339 timestamps). |
cursor, limit | Cursor-based pagination; limit defaults to 50 and is capped at 200 per page. |
Fetch a single event with GET /events/{id}. Paging uses an opaque cursor rather than an offset, so a continuously growing log paginates deterministically even when several events share a timestamp.
GET /events?resource_type=workflow&action=workflow.published&from=2026-05-01T00:00:00Z&limit=50PII-access auditing
Decisions made about personal data carry their own audit projection. GET /audit/pii-access (restricted to system_admin) returns every pii_access and pii_data_export event for the tenant, with the event's metadata flattened into top-level fields — entity type, boundary, purpose category, decision (Tokenize, PartialMask, Redact, or PassThrough), integration type, and byte count for exports. Filter by entity_type, boundary, purpose_category, integration_type, decision, actor_id, and time range. This is the dedicated surface for privacy and compliance review; the general audit feed above is for operational change history.
Retention
Audit events are kept for a configurable window per tenant (365 days by default) and removed by a background retention process once they age out. There is no manual delete — immutability is the point.
Reviewing conversations
The audit trail tells you what changed. To see what a chat AI Employee actually did — what it said to a user, what the user thought of it — review the conversation itself.
Admin-gated. Conversation review is restricted to system_admin and builder_admin. It is available for chat AI Employees, where there are conversations to review.
The conversation list
Each AI Employee has a conversation-review surface that lists the conversations it has handled, with columns for the conversation title, channel, resolution status, user satisfaction, reviewer comment count, and date. Filter the list by:
- Resolution status — the reviewer-assigned state of the conversation (below).
- Quick filters — narrow to the conversations that most need attention.
- Sentiment — filter by the end-user feedback signal (thumbs up/down) attached to the conversation.
Selecting a conversation opens a detail drawer with the full transcript.
Inside a conversation
The detail drawer shows the conversation end to end:
- Transcript — every message in order, user and AI Employee, the way the user saw it.
- User feedback — the thumbs-up / thumbs-down a user left on a message, surfaced inline and rolled up into the conversation's satisfaction score. Satisfaction is computed live from the messages, so it always reflects the current feedback.
- Reviewer comments — notes a reviewer attaches to a specific message in the transcript, visible to other reviewers. Add a comment to flag a bad answer, delete one you no longer need.
- Resolution status — a reviewer-set label on the whole conversation. Set it from the drawer to track triage: New, Opened, Resolved, Needs improvement, or Needs clarification.
This is the qualitative half of monitoring: the metrics dashboard tells you what moved; the conversations tell you why. A drop in satisfaction sends you here to read the conversations behind it, mark the bad ones Needs improvement, and feed concrete examples back into your evaluation dataset.
How auditing fits the lifecycle
- Before launch — confirm the AIE's change history on the Activity rail and check that access changes (
workflow_access.updated,user.role_changed) are correct. See the Launch checklist. - After launch — watch metrics, then use conversation review to read the conversations behind any movement and the audit trail to confirm who changed what.
- For compliance — export the tenant audit log to CSV, or pull the PII-access feed for a privacy review.
What's next
- Launch checklist — the pre-launch readiness checks, including the audit and access review.
- Launching and monitoring — publish a version and watch the metrics that conversation review explains.
- Debug logs — the execution-level trace behind a single run.