Delete an evaluation test case

Evaluations let you check an AI agent's behavior before and after you change it, the way unit tests check code. Through these endpoints you can build and run them from a script or a CI job. The model has four levels:

Writing test cases

expectedOutput is a rubric for the judge, such as "Apologizes and offers a full refund", not an exact string to match. toolCalls lists tools the agent must call, as { id, name }. A case with neither always passes, so give each case at least one. To replay a real conversation, send conversationId and the customer's messageId together when you create the case; the agent then gets the conversation up to that message as context, returned as multiTurnData. Those two fields can't be changed later. resumeNode starts the agent from a specific step instead of the beginning. Test case names must be unique within an evaluation.

Runs and scoring

POST /v1/ai/evaluations/{evaluationId}/run starts a run against the agent's current draft and returns it with status: "pending"; it moves to running and then completed. runsPerTestCase (up to 25) repeats each case to expose inconsistent answers. Each result passes only if the agent didn't error (status isn't error), the judge found that the answer met expectedOutput (assertionResult.inputResult.met, with an explanation and confidence), and every expected tool was called (assertionResult.toolCallsResult[].toolCalled). assertionResult.assertionFailed is true when any check failed. A test case's score is the percentage of its results that passed, and the run's score is the overall percentage.

During a run, tools with side effects, such as sending messages, changing data or HTTP calls other than GET, are recorded but not executed, so you can check that the agent would have issued a refund without issuing one.

Which endpoint to use

Starting a run fails with 404 if the evaluation doesn't exist or has no test cases, and with 409 if a run of that evaluation is already in progress, so poll the current run until it's completed before starting another. When you update a test case with PUT, send input, resourceId and resourceType every time; leaving out toolCalls keeps the existing ones, and [] clears them. All lists are paginated with page and pageSize, and every endpoint needs the AI Agents feature on your plan. Reading only needs read access to AI agents, but creating, changing and running evaluations requires an admin role.