Setting Up Agent QA

This section walks you through configuring Agent QA from scratch. By the end, you will have a fully working QA system evaluating your customer service calls automatically.

2.1 Prerequisites

Before starting setup, prepare three sets of files. The quality of these inputs directly determines the accuracy of your evaluations.

Golden Call Recordings

Representative audio files that cover your typical call scenarios.

Label Data CSV

A spreadsheet containing your human reviewers' Pass/Fail/N/A grades for each golden call recording.

Example:

UCID,Transcript,Greeting,Customer Verification,Issue Resolution,Empathy,Hold Time
call-001,"Good morning, thank you for calling...",Pass,Pass,Fail,Pass,N/A
call-002,"Yeah what do you need?",Fail,Pass,Pass,Fail,Pass
call-003,"Hello, this is Sarah from Support...",Pass,N/A,Pass,Pass,Pass

QA Form

2.2 Create Your AI Employee

1. Log in at app.ema.co
2. Navigate to AI Employees in the left sidebar
3. Click Create on "Pre-built AI Employee"
4. Select the Agent QA template from the catalog
5. Enter a Name (e.g., "QA Assistant - Customer Service")
6. Enter a Description (e.g., "Evaluates all inbound customer service calls against our 25-parameter QA scorecard")
7. Select the Language for evaluations (this determines the language used in scorecard reasoning and insights)
8. Click Continue

Your AI Employee is created in Disabled state. You must complete all configuration steps before enabling it.

After creation, you will see five tabs:

2.3 Configure QA Parameters with Auto-Learning

Auto-Learning is the fastest and most accurate way to configure your evaluation rules. It uses your golden dataset to generate, calibrate, and iteratively refine parameter instructions until they align with your human reviewers' grades.

Upload Your Files

1. Open your AI Employee and click the Configuration tab
2. Scroll to QA Parameters and click Setup QA Parameters
3. Upload all files:

Validation Before Auto-Learning Can Start

Run Auto-Learning

1. After all files are uploaded and validated, click Generate Parameters
2. Auto-Learning begins processing through multiple stages

What you see during processing:

• A split-panel view: the left pane displays the parameters list with per-iteration accuracy data, while the right pane shows an overall accuracy chart tracking improvement across iterations
• You can click Run in background to continue configuring other settings while it runs
• If processing fails, a Retry button appears to restart the pipeline
• Processing typically takes 15 to 45 minutes depending on the number of golden recordings

What happens behind the scenes: The system runs up to 3 improvement iterations by default, stopping early if accuracy converges (i.e., improvement between consecutive iterations falls below a 1% threshold). Parameters with accuracy below 90% are targeted for instruction rewriting in each iteration.

After the initial run completes, you can click Deep Optimize to run additional iterations (up to 20) if accuracy on certain parameters is below your target. Deep Optimize also uses convergence-based early stopping.

Review and Approve Parameters

After Auto-Learning completes:

1. Click Review Parameters
2. For each parameter, review:

3. Edit any parameter by clicking on it -- you can modify instruction text, weight, criticality, or mechanism
4. All parameters are selected by default. Deselect any you want to discard, then click Approve Parameters

Understanding Accuracy Results

Adding Parameters Manually

You can also add parameters without Auto-Learning:

1. In Configuration > QA Parameters, click Add
2. Define the parameter name, category, weight, criticality, mechanism, pass/fail instruction, and N/A instruction
3. Click Save

2.4 Add a Knowledge Base

Setting Up Data Sources

1. In the Configuration tab, scroll to Data Sources
2. Click Add and select your source type:

3. Create a folder name to organize your documents (e.g., "Return Policies," "Product Guides")
4. Upload or connect your files
5. Optionally, assign tags to documents for precise filtering when connecting folders to specific parameters

Supported File Formats

Scanned documents pass through OCR automatically. Diagrams are not interpreted -- add text captions for important visual content.

2.5 Add Integrations

Available Integrations

Setup Steps

1. In the Configuration tab, scroll to Integrations
2. Click Add Integration and select the type
3. Enter credentials -- API URL, API Key or OAuth credentials, integration-specific settings
4. Click Test Connection to verify connectivity
5. Configure field mappings -- specify which data fields to retrieve and how to match them to call interactions
6. Click Save
7. Connect the integration to your Action Based parameters in the QA Parameters section

2.6 Configure Contact Reasons

Contact reasons categorize why customers are calling. This classification powers the analytics in your Metrics dashboard.

The Three-Level Hierarchy

Configure Non-Resolution Reasons

Non-resolution reasons categorize why an issue was not resolved. The default L1 categories are:

Where to Make Changes

2.7 Set Up Permissions

Agent QA uses role-based access control with four roles. See Roles and Permissions for the full reference.

1. Click the Permissions tab in your AI Employee
2. Click Add User
3. Enter the user's email address
4. Select the appropriate role
5. Click Send Invitation

2.8 Enable and Send Your First Call

Pre-Enable Checklist

• QA parameters configured and approved (via Auto-Learning or manually)
• Knowledge base connected and documents indexed (if using Knowledge Verified parameters)
• Integrations added and connections tested (if using Action Based parameters)
• Contact reasons configured at all three levels (L1, L2, L3)
• Non-resolution reasons configured (L1 and L2)
• Permissions assigned and invitations sent

Enable Your AI Employee

1. Navigate to the Configuration tab
2. Click Save changes if any pending changes are indicated
3. Click the Enable toggle at the top of the page and confirm

Verify with a Test Call

Upload a test call via the API to verify end-to-end processing:

API endpoints:

• Audio Files: POST https://api.ema.co/api/v1/external/upload/file
• Transcript File: POST https://api.ema.co/api/v1/external/upload/transcript

Required headers:

Required form fields:

Example (Python):

import requests

API_URL = "https://api.ema.co/api/v1/external/upload/file"
PERSONA_ID = "your-persona-id"
BEARER_TOKEN = "your-jwt-token"

headers = {
 "x-persona-id": PERSONA_ID,
 "Authorization": f"Bearer {BEARER_TOKEN}",
}

data = {
 "resource_id": "test-call-001",
 "agent_id": "agent-uuid",
 "agent_email": "[email protected]",
 "channels": "1",
 "timestamp": "2024-01-15 14:30:00",
 "is_golden_data": "false",
}

with open("test_call.wav", "rb") as f:
 files = {"file": ("test_call.wav", f, "audio/wav")}
 response = requests.post(API_URL, headers=headers, data=data, files=files)

print(response.status_code, response.text)

Expected response: HTTP 200 with a confirmation message.

Verify the Results

1. Wait 2 to 5 minutes for processing
2. Navigate to the Audit tab
3. Your test interaction should appear with a status of "QA completed"
4. Click View Details to review the scorecard, transcript, and reasoning

Troubleshooting a Failed Test Call