Create an evaluation test case

post/v1/ai/evaluation-testcases
Requires any one of these rolesAbout roles
org.permission.assistant.create
Legacyorg.admin.apps.write

The API key or user must hold at least one of them. Legacy roles still work; prefer the others for new keys.

Creates a new test case for an evaluation.

About this resource

AI evaluations

An evaluation is a test suite for one AI agent. It holds test cases, each a customer message plus a description of a good answer; running the evaluation sends every case to the agent, has an AI judge mark each answer pass or fail, and gives the agent a score from 0 to 100.

Evaluations let you check an AI agent's behavior before and after you change it, the way unit tests check code. Through these endpoints you can build and run them from a script or a CI job. The model has four levels:

Object What it is
Evaluation A named suite for one AI automation, identified by automationId.
Test case One scenario in the suite: the customer's input, the customer whose data the agent's tools see (resourceType is always customer, with resourceId), and what to check: expectedOutput and toolCalls.
Run One execution of the whole suite, with a status, an overall score, and scoreByTestCase.
Test case result One attempt at one test case in a run: the agent's output, the tools it called, and assertionResult, the verdict.

Writing test cases

expectedOutput is a rubric for the judge, such as "Apologizes and offers a full refund", not an exact string to match. toolCalls lists tools the agent must call, as { id, name }. A case with neither always passes, so give each case at least one. To replay a real conversation, send conversationId and the customer's messageId together when you create the case; the agent then gets the conversation up to that message as context, returned as multiTurnData. Those two fields can't be changed later. resumeNode starts the agent from a specific step instead of the beginning. Test case names must be unique within an evaluation.

Runs and scoring

POST /v1/ai/evaluations/{evaluationId}/run starts a run against the agent's current draft and returns it with status: "pending"; it moves to running and then completed. runsPerTestCase (up to 25) repeats each case to expose inconsistent answers. Each result passes only if the agent didn't error (status isn't error), the judge found that the answer met expectedOutput (assertionResult.inputResult.met, with an explanation and confidence), and every expected tool was called (assertionResult.toolCallsResult[].toolCalled). assertionResult.assertionFailed is true when any check failed. A test case's score is the percentage of its results that passed, and the run's score is the overall percentage.

During a run, tools with side effects, such as sending messages, changing data or HTTP calls other than GET, are recorded but not executed, so you can check that the agent would have issued a refund without issuing one.

Which endpoint to use

To Call
List an agent's evaluations GET /v1/ai/automations/{automationId}/evaluations
Create, read, rename or delete an evaluation POST /v1/ai/evaluations, GET, PUT, DELETE /v1/ai/evaluations/{evaluationId}
List test cases, filtered by evaluationIds GET /v1/ai/evaluation-testcases
Create, read, update or delete a test case POST /v1/ai/evaluation-testcases, GET, PUT, DELETE /v1/ai/evaluation-testcases/{testCaseId}
Start a run POST /v1/ai/evaluations/{evaluationId}/run
Check a run's status and score GET /v1/ai/evaluation-runs/{runId}, or GET /v1/ai/evaluation-runs with evaluationIds for history
See each result and why it failed GET /v1/ai/evaluation-testcase-results with evaluationRunIds, or GET /v1/ai/evaluation-testcase-results/{resultId}

Starting a run fails with 404 if the evaluation doesn't exist or has no test cases, and with 409 if a run of that evaluation is already in progress, so poll the current run until it's completed before starting another. When you update a test case with PUT, send input, resourceId and resourceType every time; leaving out toolCalls keeps the existing ones, and [] clears them. All lists are paginated with page and pageSize, and every endpoint needs the AI Agents feature on your plan. Reading only needs read access to AI agents, but creating, changing and running evaluations requires an admin role.

RelatedAI and automations APIAI automationsCustomers

Request body

application/jsonrequired
namestringrequired

Name of the test case

max length 100Example "Refund request scenario"
evaluationIdstringrequired

Associated evaluation ID

Example "507f1f77bcf86cd799439012"
inputstringrequired

Test case input data

max length 10000Example "I want a refund for my order."
conversationIdstring

Source conversation ID for multi-turn context

Example "507f1f77bcf86cd799439014"
messageIdstring

Source message ID for multi-turn context

Example "507f1f77bcf86cd799439015"
resourceIdstringrequired

Associated resource ID

Example "507f1f77bcf86cd799439013"
resourceTypestringrequired

Resource type (currently only customer)

One of"customer"Example "customer"
array<object>

Tool calls expected during execution

resumeNodestring

Node ID to resume execution from

Example "node-2"
expectedOutputstring

Expected output for the test case

max length 1024Example "Sure, I can help with that. Can you share your order ID?"

Responses

Created

objectTestCase