Setting Up Agent QA

This section walks you through configuring Agent QA from scratch. By the end, you will have a fully working QA system evaluating your customer service calls automatically.

2.1 Prerequisites

Before starting setup, prepare three sets of files. The quality of these inputs directly determines the accuracy of your evaluations.

Golden Call Recordings

Representative audio files that cover your typical call scenarios.

Label Data CSV

A spreadsheet containing your human reviewers' Pass/Fail/N/A grades for each golden call recording.

Example:

UCID,Transcript,Greeting,Customer Verification,Issue Resolution,Empathy,Hold Time
call-001,"Good morning, thank you for calling...",Pass,Pass,Fail,Pass,N/A
call-002,"Yeah what do you need?",Fail,Pass,Pass,Fail,Pass
call-003,"Hello, this is Sarah from Support...",Pass,N/A,Pass,Pass,Pass

QA Form

2.2 Create Your AI Employee

1. Log in at app.ema.co
2. Navigate to AI Employees in the left sidebar
3. Click Create on "Pre-built AI Employee"
4. Select the Agent QA template from the catalog
5. Enter a Name (e.g., "QA Assistant - Customer Service")
6. Enter a Description (e.g., "Evaluates all inbound customer service calls against our 25-parameter QA scorecard")
7. Select the Language for evaluations (this determines the language used in scorecard reasoning and insights)
8. Click Continue

Your AI Employee is created in Disabled state. You must complete all configuration steps before enabling it.

After creation, you will see six tabs:

2.3 Import Your Data

The Imports tab is where you bring call recordings and transcripts into your AI Employee for evaluation. It syncs automatically, showing "Synced a few seconds ago" or "Not synced yet" -- click Refresh at any time to pull the latest status without waiting.

Imports and New Import

The Imports sub-tab lists every import job -- its name, call window, status, progress, and timestamps. Click New import to start one, choosing from:

Connecting Genesys Cloud

Instead of manually uploading files, you can connect Genesys Cloud to pull matching recordings automatically:

1. Choose Connect to Genesys from New import
2. Pick a language and date range
3. Submit -- the job is queued immediately; you don't need to wait for it to finish before doing anything else

Behind the scenes, Ema finds every matching call and matches each call's agent identifier to a known employee using your agent-mapping configuration (see Agent Mapping below). If some agents can't be matched, the job pauses in an Awaiting your input state until you add the missing mapping or skip those calls, then click Resolve Agent to resume the job -- it does not resume automatically. Once resolved, recordings are fetched and handed off for processing at a fair, rate-limited pace, so one large import can't crowd out other work. QA results appear in the Audit tab as usual.

Agent Mapping

The Agent Mapping sub-tab maps the agent identifier used by your source system (e.g., Genesys, or your metadata CSV) to a known Ema employee, with an effective-from/effective-to validity window per mapping (so the same source ID can point to different employees over time). To bulk-update mappings, use Download mapping to export the current set, edit it, then Upload mapping -- a wizard walks you through choosing the file, matching columns, picking additive or full-replace mode, and reviewing the diff before it's applied.

2.4 Configure QA Parameters with Auto-Learning

Auto-Learning is the fastest and most accurate way to configure your evaluation rules. It uses your golden dataset to generate, calibrate, and iteratively refine parameter instructions until they align with your human reviewers' grades.

Upload Your Files

1. Open your AI Employee and click the Configuration tab
2. Scroll to QA Parameters and click Setup QA Parameters
3. Upload all files:

Previously uploaded files are visible in this screen after a page reload or in a new session -- the setup shows your most recent upload, not a full history. Reopening the setup pre-populates those files by default; to run with different data, delete the previous files and re-upload. A persona whose Auto-Learning has never run opens with no pre-populated state.

Validation Before Auto-Learning Can Start

Run Auto-Learning

1. After all files are uploaded and validated, click Generate Parameters
2. Auto-Learning begins processing through multiple stages

What you see during processing:

• A split-panel view: the left pane displays the parameters list with per-iteration accuracy data, while the right pane shows an overall accuracy chart tracking improvement across iterations
• You can click Run in background to continue configuring other settings while it runs
• If processing fails, a Retry button appears to restart the pipeline
• Processing typically takes 15 to 45 minutes depending on the number of golden recordings
• While a run is active, manual rule editing (add/edit/delete) is disabled with an explanatory notice -- it re-enables once the run finishes

What happens behind the scenes: The system runs up to 3 improvement iterations by default, stopping early if accuracy converges (i.e., improvement between consecutive iterations falls below a 1% threshold). Parameters with accuracy below 90% are targeted for instruction rewriting in each iteration.

After the initial run completes, you can click Deep Optimize to run additional iterations (up to 20) if accuracy on certain parameters is below your target. Deep Optimize also uses convergence-based early stopping.

Review and Approve Parameters

After Auto-Learning completes:

1. Click Review Parameters
2. For each parameter, review:

3. Edit any parameter by clicking on it -- you can modify instruction text, weight, criticality, or mechanism
4. All parameters are selected by default. Deselect any you want to discard, then click Approve Parameters

Understanding Accuracy Results

Appending New Parameters

Auto-Learning is not a one-time, one-shot setup -- you can add more rules to a persona that already has approved parameters:

1. Upload additional golden data (audio/transcripts, label data, and, if needed, an updated QA form covering only the new parameters)
2. Click Generate Parameters as before

Auto-Learning runs only for the newly added parameters -- your existing approved parameters, their results, and their pipeline state are untouched. Once the run completes, approving it merges the new parameters into your existing set; it never replaces or clears what's already there. You get the same options as a first-ever run: approve as-is, or Deep Optimize the new parameters further before approving.

Adding Parameters Manually

You can also add parameters without Auto-Learning:

1. In Configuration > QA Parameters, click Add
2. Define the parameter name, category, weight, criticality, mechanism, pass/fail instruction, and N/A instruction
3. Click Save

Bulk Delete and Reset

Select multiple parameters via checkboxes (or select all) to delete them at once -- including all of them, down to an empty parameter set. Resetting to zero shows a warning first, since any continuous-learning corrections tied to the deleted parameters are lost. From an empty set, you can rebuild via Auto-Learning (as a fresh first run) or by adding parameters manually.

2.5 Add a Knowledge Base

Setting Up Data Sources

1. In the Configuration tab, scroll to Data Sources
2. Click Add and select your source type:

3. Create a folder name to organize your documents (e.g., "Return Policies," "Product Guides")
4. Upload or connect your files
5. Optionally, assign tags to documents for precise filtering when connecting folders to specific parameters

Supported File Formats

Scanned documents pass through OCR automatically. Diagrams are not interpreted -- add text captions for important visual content.

2.6 Add Integrations

Available Integrations

Setup Steps

1. In the Configuration tab, scroll to Integrations
2. Click Add Integration and select the type
3. Enter credentials -- API URL, API Key or OAuth credentials, integration-specific settings
4. Click Test Connection to verify connectivity
5. Configure field mappings -- specify which data fields to retrieve and how to match them to call interactions
6. Click Save
7. Connect the integration to your Action Based parameters in the QA Parameters section

2.7 Configure Contact Reasons

Contact reasons categorize why customers are calling. This classification powers the analytics in your Metrics dashboard.

The Three-Level Hierarchy

Configure Non-Resolution Reasons

Non-resolution reasons categorize why an issue was not resolved. The default L1 categories are:

Where to Make Changes

2.8 Additional Configuration Tab Settings

Glossary

Upload company-specific terms and their definitions so Ema understands your organization's vocabulary -- product names, internal abbreviations, or jargon that a general-purpose model wouldn't otherwise recognize.

Call Processing

Controls how call audio is processed before it undergoes QA evaluation.

By default, audio calls are redacted with two independent passes for personally identifiable information (PII) -- rather than a single pass -- so that PII missed by one pass can still be caught by the other. Redacted values are shown as their specific category (e.g. [FIRST_NAME], [SSN]) rather than a generic placeholder.

Output Language

Sets the output language for this persona. Insights, Audit rationale, CSAT, and Coaching Reports are generated in the selected language. The transcript and rule definitions are never translated, and verbatim quotes from the transcript remain in their original language. Changes apply only to new calls processed after saving.

Configure EmaFusion

By default, Ema intelligently combines multiple large language models to give your persona the best performance. You can instead choose a specific set of models, or bring your own via API details. If you limit to specific models, choose an Optimization Priority: Auto (balances accuracy, latency, and cost by task type), Fastest, Cheapest, or Most Accurate. This setting applies to every agent in the persona by default; you can override it per agent if needed.

Data Retention

Raw call audio and raw uploaded transcripts are automatically deleted after a retention period -- 30 days by default, configurable per tenant or per persona (a persona-level setting overrides the tenant default, which overrides the platform default). Processed transcripts and evaluation records (scores, scorecards, insights) have no retention limit and are kept indefinitely.

2.9 Set Up Permissions

Agent QA uses role-based access control with five roles. See Roles and Permissions for the full reference.

1. Click the Permissions tab in your AI Employee
2. Click Add User
3. Enter the user's email address
4. Select the appropriate role
5. Click Send Invitation

2.10 Enable and Send Your First Call

Pre-Enable Checklist

• QA parameters configured and approved (via Auto-Learning or manually)
• Knowledge base connected and documents indexed (if using Knowledge Verified parameters)
• Integrations added and connections tested (if using Action Based parameters)
• Contact reasons configured at all three levels (L1, L2, L3)
• Non-resolution reasons configured (L1 and L2)
• Permissions assigned and invitations sent

Enable Your AI Employee

1. Navigate to the Configuration tab
2. Click Save changes if any pending changes are indicated
3. Click the Enable toggle at the top of the page and confirm

Verify with a Test Call

Upload a test call via the API to verify end-to-end processing:

API endpoints:

• Audio Files: POST https://api.ema.co/api/v1/external/upload/file
• Transcript File: POST https://api.ema.co/api/v1/external/upload/transcript

Required headers:

Required form fields:

Example (Python):

import requests

API_URL = "https://api.ema.co/api/v1/external/upload/file"
PERSONA_ID = "your-persona-id"
BEARER_TOKEN = "your-jwt-token"

headers = {
 "x-persona-id": PERSONA_ID,
 "Authorization": f"Bearer {BEARER_TOKEN}",
}

data = {
 "resource_id": "test-call-001",
 "agent_id": "agent-uuid",
 "agent_email": "[email protected]",
 "channels": "1",
 "timestamp": "2024-01-15 14:30:00",
 "is_golden_data": "false",
}

with open("test_call.wav", "rb") as f:
 files = {"file": ("test_call.wav", f, "audio/wav")}
 response = requests.post(API_URL, headers=headers, data=data, files=files)

print(response.status_code, response.text)

Expected response: HTTP 200 with a confirmation message.

Verify the Results

1. Wait 2 to 5 minutes for processing
2. Navigate to the Audit tab
3. Your test interaction should appear with a status of "QA completed"
4. Click View Details to review the scorecard, transcript, and reasoning

Troubleshooting a Failed Test Call