In Apex, a breaking change fails to compile or fails a test. In an agent, a breaking change deploys cleanly, demos fine, and quietly misroutes customers. The only defense is a baseline you rerun on every change.
Brightcart, an online retailer, runs a customer-service agent with topics for order status, returns and refunds, product questions, and delivery issues. It's been live for months. The refund topic is the workhorse: it verifies the order, checks the return window, and creates a refund case.
One Tuesday, a product manager asks for a small change: the agent should stop offering refunds proactively and instead try a replacement first. A developer edits one instruction on the returns topic — from "Help the customer return or refund their order" to "Always offer a replacement before discussing a refund. Only process refunds when the customer explicitly declines a replacement." Deploys to production. Demo works: ask for a refund, get offered a replacement, decline it, refund proceeds. Ticket closed.
Over the next two weeks, refund case volume drops 18%. Nobody celebrates — because customer complaints rise. Transcripts eventually show what happened: customers who said things like "the blender arrived broken, I want my money back" were now getting replacement offers, saying "no thanks, refund please", getting another replacement variant offered, and abandoning the chat. The instruction "always offer a replacement" was being applied even after an explicit decline, because "explicitly declines" competed with "always". A single sentence changed the behavior of every conversation in the topic — and nothing failed, because nothing was being tested.
For beginners: this is what makes agent changes different from code changes. Topic instructions, descriptions, action labels and the underlying model are all inputs to a probabilistic planner. Editing any of them can shift behavior anywhere in the agent — including places you didn't touch. There is no compiler to catch it. A regression suite — a fixed set of test cases with known-good results, rerun on every change — is the compiler you have to build yourself.
Brightcart already had the raw material: a 160-case test spec built the way Part 1 and Part 2 describe, in git next to the agent metadata. What they hadn't done was declare it a baseline: the suite, at 100% pass, that defines current correct behavior. The rule they adopted:
Had the rule existed on that Tuesday, the PM's change would have shipped with a diff like this, and the review conversation would have happened before production:
- utterance: the blender arrived broken, I want my money back
expectedTopic: Returns_And_Refunds
expectedActions:
- - Create_Refund_Case
+ - Offer_Replacement # intended? customer asked for money back
expectedOutcome: Acknowledges the damaged item and processes
- a refund for the order.
+ a replacement, offering a refund only if the customer declines.That comment on the diff is the whole point. A reviewer looking at this case would have asked the question the demo never surfaced: should "I want my money back" really get a replacement offer first? The PM's answer ("no — explicit refund requests skip the offer") becomes a sharper instruction and a test case, before any customer sees it.
Brightcart reruns the full suite on every change to any input the planner sees. Their list, worth stealing:
The suite only protects you if it runs without anyone remembering to run it. The sf agent test commands are plain CLI, so any CI system works. Brightcart's GitHub Actions job runs on every pull request that touches agent metadata or the spec, deploys to a dedicated testing sandbox, reruns the suite, and publishes results as JUnit so failures annotate the PR:
name: agent-regression
on:
pull_request:
paths:
- "force-app/main/default/genAiPlannerBundles/**"
- "force-app/main/default/genAiPlugins/**"
- "force-app/main/default/genAiFunctions/**"
- "specs/**"
jobs:
agent-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Salesforce CLI
run: npm install --global @salesforce/cli
- name: Authenticate to the testing sandbox
run: |
echo "${{ secrets.SF_AUTH_URL }}" > auth.txt
sf org login sfdx-url --sfdx-url-file auth.txt --alias agent-qa
rm auth.txt
- name: Deploy agent metadata and test definition
run: |
sf project deploy start --target-org agent-qa
sf agent test create \
--spec specs/brightcartServiceAgent.yaml \
--api-name Brightcart_Service_Regression \
--target-org agent-qa --force-overwrite
- name: Run the regression suite
run: |
sf agent test run \
--api-name Brightcart_Service_Regression \
--target-org agent-qa \
--wait 20 \
--result-format junit > test-results.xml
- name: Publish results
if: always()
uses: actions/upload-artifact@v4
with:
name: agent-test-results
path: test-results.xmlNotes from getting this stable in practice:
If your agent performs writes — refunds are the classic case — regression testing pairs with hard guardrails in the actions themselves: typed inputs, validation and confirmation, as covered in Governed Agent Actions, and deterministic policy transitions as in Agent Script refund guardrails. Tests catch the drift; guardrails cap the blast radius when something gets through.
Next in the series: the final layer — guardrail and red-team testing, where the "user" in your test cases is actively trying to break the agent. The full series lives on the Agentforce Testing & Evaluation track.
Sources: Test an Agent — Agentforce DX (Developer Guide) · Agentforce Testing Center (Salesforce Help) · Agentforce Testing Center (Trailhead)