Testing AI Employees

Testing and evaluation is an ongoing process for all AI Employees. This guide covers how to create refined test datasets, perform test-driven AI Employee development, and continue evaluating AI Employees that have already been deployed.

Testing New AI Employees

The testing approach varies by AI Employee type. The examples below illustrate the process for two common types:

1. Chat-based AI Employee -- A policy assistant that responds to employee queries and personalizes answers based on location and level.
2. Dashboard-based AI Employee -- A financial audit assistant that categorizes incoming documents as invoices or purchase orders, extracts values, and validates them against company rules.

Step 1: Testbed Creation and Test Prep

Whether you are creating a new AI Employee from a template or via a custom GWE workflow, begin by collecting all existing use cases in a single place.

Chat AI Employees

1. Create a diverse query testbed. Build an extensive set of queries that end users might ask. Include variations across all supported languages, synonyms, and company-specific terminology.

2. Account for follow-up queries. Create a separate testbed that outlines follow-up queries and expected responses. Cover both:

   • Follow-ups relevant to the previous query (to verify context retention).
   • Follow-ups irrelevant to the previous query (to rule out context contamination).

3. Account for different user types. If user parameters matter for personalization (e.g., location, role), ensure your testbed includes queries from each user type.

4. Include out-of-scope cases. Flag deprioritized, inappropriate, or irrelevant queries and set up the AI Employee to respond with a boilerplate disclaimer explaining they are out of scope.

5. Create golden response outputs. Capture all the key elements of a good response: required facts, tone, and disclaimers. Since agentic responses are non-deterministic, the AI Employee response will not always match the golden response exactly. You can either:

   • Have a human evaluate whether the response covers key points.
   • Use an LLM evaluator with a golden prompt to check coverage.

Dashboard AI Employees

1. Create a varied testbed of inputs. For example, for a document processing AI Employee, include a variety of invoices and purchase orders.

2. Represent all document formats and complexities. Include image-heavy documents, large documents, and handwritten, blurry, or scanned documents where applicable. Exclude formats that are not relevant to your use case.

Both Types

• Account for HITL use cases. Mark test cases where Human-In-The-Loop is required and note potential inputs from the user at each step.
• Prepare sandbox environments. If you cannot use production app integrations for testing, have a sandbox version with similar data ready.

Step 2: Progressive Testing During Development

Build and test the workflow incrementally rather than all at once.

1. Tackle a few test cases at a time. Progressively build an AI Employee that can address your entire testbed. Different team members can build different parts of the workflow in isolation and consolidate later.

   For example, with a policy assistant, start by adding support for payroll-related queries only. Hand over this v1 to payroll SMEs for testing while you build out support for other categories. Once payroll queries are answered satisfactorily, add an intent categorizer and test whether it accurately differentiates between payroll, vacation, and medical queries.

2. Publish intermediate outputs for debugging. Use intermediate responses to inspect each step of the workflow individually. You can:

   • Add "Logging" steps using the Fixed Response agent combined with workflow branching via Trigger When to detect which arm of the workflow was triggered.
   • Publish the output of helper agents (e.g., a Query Augmenter) to verify they work correctly before building the next steps.
   • Unpublish intermediate outputs once you are satisfied and move on.

3. Use Show Work to debug responses. Show Work logs display the agents invoked, steps taken, and error messages for each response. See the section below.

Using Show Work

Show Work logs help demonstrate the steps taken to respond to each query. They provide a log of agents invoked, steps taken by each agent, and error messages if any agent failed.

• For chat AI Employees: View Show Work logs in the Ema webapp chat. If building an embeddable chatbot, go to the GWE Canvas and click "Use AI Employee" to start a webapp-based chat (the test chatbot does not display Show Work logs since it emulates the end-user experience).
• How to use Show Work:
  1. Inspect which category was chosen to verify the correct workflow branch was selected.
  2. Check which agent might be causing quality issues (e.g., zero search results, or an oversized model causing slowness).
  3. In cases of complete failure, view the last agent that ran and any error messages it produced.

Step 3: User Acceptance Testing (UAT)

Before launching to production, run a full round of end-to-end testing.

1. Go through the entire testbed. Impersonate all kinds of users with the same AI Employee.
2. Mock production scale. Ensure the AI Employee will not buckle under production volumes.
3. Remove all debug artifacts. Remove intermediate outputs to ensure the experience matches what an actual end user will see.
4. Recruit actual testers. Nominate a few real users as testers with minimal instructions to mimic actual end-user behavior. Their feedback is often more useful for improving onboarding and training material than for improving the product itself.

Testing Existing AI Employees

Testing existing AI Employees is more challenging because live users may be impacted by regressions. All the steps above apply, along with these additional strategies:

1. Create a copy of the AI Employee. Use the clone feature (see Seamless Upgrade) to maintain a separate test version that is always slightly ahead of the production version.

2. Test isolated workflows. If you plan to add a new branch, test it in isolation in a new AI Employee, then merge it into the main AI Employee once you are satisfied.

Quick Tips

1. Save workflow copies. Store copies of past workflows so you can easily inspect or roll back if you see regressions. Download the persona config from the browser inspector (copy the response of the "Get personas" API call) or copy the config from the code view in the GWE visual editor.

2. Track changes and impact. Keep an annotated log of changes and corresponding test results so you can correlate changes with outcomes.

3. Keep evaluations objective. Use blind evaluations between the AI Employee and a human performing the same role. Users frequently overestimate human accuracy; benchmarking against humans establishes relative accuracy.

4. Launch progressively. It is acceptable to launch test cases where the AI Employee performs well while letting it take a back seat on complex cases. For those cases, configure HITL for human approval or provide a disclaimer so users know the AI Employee may be less reliable.