QA Merge Agent

The QA Merge Agent deduplicates and merges question/answer (Q&A) pairs. It embeds each question, clusters near-duplicates by semantic similarity, and uses an LLM to decide which clustered pairs are genuinely the same question — marking duplicates rather than silently dropping them, so the original rows stay traceable. It runs in two phases: first it self-deduplicates the incoming pairs, then it deduplicates them against an optional set of existing pairs.

It belongs to the Custom Agents group: a set of pre-built, externally dispatched agents you select by ID instead of configuring an LLM loop yourself. Like every agent in this group, it runs as a Custom agent (custom_agent) — the agent service forwards the node's inputs to the custom-action service, which executes the agent and returns its output. See Custom Agent for how the custom_agent dispatch mechanism works.

When to use it

Use the QA Merge Agent to clean up a generated or extracted Q&A set before it lands in a knowledge base — removing near-duplicate questions, consolidating overlapping answers, and (optionally) reconciling duplicates that came from translation. It commonly runs downstream of an Extraction agent and the Multilingual Translation Agent: translation produces localized copies, and this agent reconciles them.

How it works

The agent processes pairs in two phases:

1. Self-deduplication — embeds and clusters the incoming pairs by question similarity, then asks the LLM to confirm which clustered pairs are truly duplicates of one another.
2. Deduplication against existing — compares the survivors against any existing_qa_json you supply, so newly generated pairs that already exist are flagged.

Two deduplication strategies are available, selected by dedup_across_languages:

• Off (default) — clusters by embedding similarity, then splits clusters by language, so the agent only deduplicates within the same language. It sets the is_duplicate and duplicate_question fields.
• On — uses translation-family relationships to group, keeps one item per language for the same question/answer, and additionally sets the is_translated and is_duplicate_across_languages fields.

Configuration

Add the agent to a workflow as a Custom agent and pin it by ID in agent_config.type_config:

{ "custom_agent_id": "qa_merge_agent" }

It reads the following input keys.

Output

The agent returns an object with a single rows key containing every processed Q&A pair — both the unique pairs and the ones marked as duplicates (duplicates are flagged, not removed, so you can audit the merge). Each row carries:

A downstream node reads the result via {{<node_id>.output.rows}}.

Example

A node configured as:

{ "custom_agent_id": "qa_merge_agent" }

with these inputs:

{
  "extraction_columns": "[{\"question\": \"How do I reset my password?\", \"answer\": \"Use the reset link.\", \"language_code\": \"en-US\"}, {\"question\": \"How can I reset my password?\", \"answer\": \"Click the reset link.\", \"language_code\": \"en-US\"}]",
  "similarity_threshold": "0.91"
}

returns both pairs, with the second flagged as a duplicate of the first:

{
  "rows": [
    {
      "question": "How do I reset my password?",
      "answer": "Use the reset link.",
      "language_code": "en-US",
      "is_duplicate": false,
      "duplicate_question": ""
    },
    {
      "question": "How can I reset my password?",
      "answer": "Click the reset link.",
      "language_code": "en-US",
      "is_duplicate": true,
      "duplicate_question": "How do I reset my password?"
    }
  ]
}

What's next

• Custom Agent — how the custom_agent dispatch this agent runs on works.
• Multilingual Translation Agent — produces the localized rows this agent reconciles.
• Agent Reference overview — the agent-type model, catalog, and shared execution engine.