Agentforce Testing Center: Batch-Test Your Agent Before Go-Live

The scenario

Meridian Trust Bank, a retail bank, is three weeks from launching an account-servicing agent on its customer portal. The agent has six topics: balance inquiries, card services, statements and documents, payments and transfers, dispute intake, and branch information. The build looks done. Demos go well. Then the delivery lead asks the question that decides go-live: "If a real customer types a real question, does it land on the right topic?"

Nobody knows. The team has clicked through maybe thirty conversations by hand. The bank's live-chat history has thousands. So the team pulls 200 real customer questions from last quarter's chat transcripts, tags each with the topic and action it should trigger, and runs them all as a batch. That batch is the go-live gate. This post walks through exactly how they do it.

For beginners: Agentforce Testing Center is Salesforce's built-in tool for testing agents at scale. Instead of typing one utterance at a time into the preview panel, you upload a set of test cases — an utterance plus what you expect to happen — and run them as a batch against your agent. Each case is evaluated on whether the right topic was selected, the right actions fired, and the response was reasonable. It lives in Agentforce Studio, and the same tests can be created and run from the Salesforce CLI.

Step 1 — Build the test set from real questions

The biggest quality lever is where the utterances come from. Meridian did not invent test questions in a meeting room. They exported chat transcripts, deduplicated, and kept 200 questions customers actually asked — spelling mistakes included. "wat is my balnce" is a valid test case. Your customers will type it.

For every utterance, the team recorded three expectations:

• Expected topic — which topic the planner should select. This is the routing assertion, and for this milestone it's the one that matters most.
• Expected actions — which action (or sequence of actions) the agent should invoke inside that topic.
• Expected outcome — a plain-language description of a good response, used to judge the reply itself.

In source control this lives as a YAML test spec — the format used by Agentforce DX. Three of Meridian's 200 cases:

description: Routing validation for the account-servicing agent
subjectType: AGENT
subjectName: Account_Servicing_Agent
testCases:
  - utterance: what's the balance on my everyday account
    expectedTopic: Account_Balance_Inquiry
    expectedActions:
      - Get_Account_Balance
    expectedOutcome: States the current balance of the customer's
      account and offers further help.

  - utterance: my card got swallowed by the ATM on King Street
    expectedTopic: Card_Services
    expectedActions:
      - Block_Card
      - Order_Replacement_Card
    expectedOutcome: Blocks the card, confirms a replacement is
      ordered and tells the customer when it will arrive.

  - utterance: i dont recognise a payment of 49.99 from yesterday
    expectedTopic: Dispute_Intake
    expectedActions:
      - Create_Transaction_Dispute
    expectedOutcome: Opens a dispute for the transaction and explains
      the next steps without promising a refund.

Step 2 — Create the test in the org

The spec is local YAML. To run it, it has to exist in the org as a test definition (the metadata type behind it is AiEvaluationDefinition). Two ways to get there:

• Testing Center UI — in Agentforce Studio, open Testing Center, create a new test, pick the agent, and add test cases. You can bulk-upload cases from a CSV instead of typing them one by one — exactly what you want with 200 rows.
• Salesforce CLI — the route Meridian took, because the spec is in git and the pipeline needs it later anyway.

# Scaffold a spec interactively (or hand-write it like above)
sf agent generate test-spec

# Push the spec into the org as a runnable test definition
sf agent test create \
  --spec specs/accountServicingRouting.yaml \
  --api-name Account_Servicing_Routing_Tests \
  --target-org meridian-uat

After this, the same test is visible in Testing Center in the org — admins see and run in the UI what developers manage in source control. One test definition, two audiences. That mattered at Meridian: the service-cloud admin owned topic descriptions, and she could rerun the suite herself after every wording change without asking a developer.

Step 3 — Run the batch

A batch run executes every case against the agent in a synthetic session — no live customers, no records touched beyond what your actions do in the sandbox. From the CLI:

# Run and wait up to 10 minutes for completion
sf agent test run \
  --api-name Account_Servicing_Routing_Tests \
  --target-org meridian-uat \
  --wait 10

# Or fire-and-forget, then fetch results by job id
sf agent test run --api-name Account_Servicing_Routing_Tests --target-org meridian-uat
sf agent test results --job-id 4KBxx0000000123 --target-org meridian-uat --result-format json

In the UI it's one button: open the test in Testing Center and run it. Either way you get the same evaluation per case: did the selected topic match, did the invoked actions match, and how did the actual response measure against the expected outcome. The response check is judged by a model rather than a string comparison — more on how that works, and how to build your own deeper quality metrics, in Part 3.

Step 4 — Read the results (this is where the value is)

First run: 171 of 200 passed. That number alone is not the finding. The finding is in the failure pattern. Sorting the failures by expected topic, Meridian saw:

• 19 failures clustered on card questions. Utterances like "there's a charge on my card I don't recognise" were routed to Card_Services, but the expectation was Dispute_Intake. Root cause: the Card Services topic description said it handles "any question about the customer's card". Too greedy. The planner believed it.
• 6 failures on statements. "Can I get proof of my account for a visa application?" went to branch information. The statements topic never mentioned letters or proofs — a genuine coverage gap, not a wording problem.
• 4 failures were bad expectations. The transcript tagger had labeled "how do I close my account" as a servicing question, but closing an account was deliberately out of scope for launch. The correct behavior is escalation. The test set was wrong; the agent was right.

Three failure types, three different fixes:

1. Rewrite the Card Services topic description to explicitly exclude unrecognised charges and point disputes at the dispute topic ("classification descriptions are instructions to the planner" — write them with the same care as code).
2. Add a "proof of account / interim statement" action to the statements topic.
3. Fix the four mislabeled test cases and add an explicit expectation that out-of-scope requests escalate.

Second run: 196 of 200. The remaining four were genuinely ambiguous one-line utterances ("it's not working") where even a human agent would ask a clarifying question — the team asserted clarification behavior instead of a topic and moved on. That 200-case suite, green, became the sign-off artifact for go-live. Not a demo. Evidence.

What Meridian's team learned

• Real utterances beat invented ones. Nobody in the project team would have written "wat is my balnce" or the visa-letter question. The transcripts did.
• Most failures are topic-description failures. The fix is usually words, not code. Batch testing tells you which words.
• Expect to fix the test set too. A few percent of your expectations will be wrong. That's normal and useful — it forces the team to decide what correct behavior actually is.
• A green batch is a gate you can rerun. Every future prompt tweak, topic change, or model update gets judged against the same 200 questions.

This post covered running the machine. The next question is what you feed it: 200 utterances are only as good as their variety. Next in the series: designing test utterances that actually find bugs — paraphrases, edge cases, adversarial phrasing and multilingual inputs. The full series lives on the Agentforce Testing & Evaluation track, and if you're still getting oriented in the platform itself, start with Agentforce 360, explained for developers.

Sources: Agentforce Testing Center (Salesforce Help) · Test an Agent — Agentforce DX (Developer Guide) · Agentforce Testing Center (Trailhead)