# Update an evaluation

> PUT /v1/ai/evaluations/{evaluationId} · AI and automations API

Source: https://developer.kustomer.com/api/ai/update-an-evaluation

```http
PUT https://{orgname}.api.kustomerapp.com/v1/ai/evaluations/{evaluationId}
Authorization: Bearer <API key>
```

## Authorization

Requires any one of these roles: `org.permission.assistant.update`; or legacy `org.admin.apps.write`. The API key or user must hold at least one of them. Legacy roles predate permission sets and still work.

Updates an existing evaluation.

## About AI evaluations

An evaluation is a test suite for one AI agent. It holds test cases, each a customer message plus a description of a good answer; running the evaluation sends every case to the agent, has an AI judge mark each answer pass or fail, and gives the agent a score from 0 to 100.

Evaluations let you check an AI agent's behavior before and after you change it, the way unit tests check code. Through these endpoints you can build and run them from a script or a CI job. The model has four levels:

| Object | What it is |
| --- | --- |
| Evaluation | A named suite for one [AI automation](https://developer.kustomer.com/api/ai#automations), identified by `automationId`. |
| Test case | One scenario in the suite: the customer's `input`, the customer whose data the agent's tools see (`resourceType` is always `customer`, with `resourceId`), and what to check: `expectedOutput` and `toolCalls`. |
| Run | One execution of the whole suite, with a `status`, an overall `score`, and `scoreByTestCase`. |
| Test case result | One attempt at one test case in a run: the agent's `output`, the tools it called, and `assertionResult`, the verdict. |

#### Writing test cases

`expectedOutput` is a rubric for the judge, such as "Apologizes and offers a full refund", not an exact string to match. `toolCalls` lists tools the agent must call, as `{ id, name }`. A case with neither always passes, so give each case at least one. To replay a real conversation, send `conversationId` and the customer's `messageId` together when you create the case; the agent then gets the conversation up to that message as context, returned as `multiTurnData`. Those two fields can't be changed later. `resumeNode` starts the agent from a specific step instead of the beginning. Test case names must be unique within an evaluation.

#### Runs and scoring

`POST /v1/ai/evaluations/{evaluationId}/run` starts a run against the agent's current draft and returns it with `status: "pending"`; it moves to `running` and then `completed`. `runsPerTestCase` (up to 25) repeats each case to expose inconsistent answers. Each result passes only if the agent didn't error (`status` isn't `error`), the judge found that the answer met `expectedOutput` (`assertionResult.inputResult.met`, with an `explanation` and `confidence`), and every expected tool was called (`assertionResult.toolCallsResult[].toolCalled`). `assertionResult.assertionFailed` is `true` when any check failed. A test case's score is the percentage of its results that passed, and the run's `score` is the overall percentage.

During a run, tools with side effects, such as sending messages, changing data or HTTP calls other than `GET`, are recorded but not executed, so you can check that the agent would have issued a refund without issuing one.

#### Which endpoint to use

| To | Call |
| --- | --- |
| List an agent's evaluations | [`GET /v1/ai/automations/{automationId}/evaluations`](https://developer.kustomer.com/api/ai/list-evaluations-for-an-automation) |
| Create, read, rename or delete an evaluation | [`POST /v1/ai/evaluations`](https://developer.kustomer.com/api/ai/create-an-evaluation), [`GET`](https://developer.kustomer.com/api/ai/get-evaluation-by-id), [`PUT`](https://developer.kustomer.com/api/ai/update-an-evaluation), [`DELETE /v1/ai/evaluations/{evaluationId}`](https://developer.kustomer.com/api/ai/delete-an-evaluation) |
| List test cases, filtered by `evaluationIds` | [`GET /v1/ai/evaluation-testcases`](https://developer.kustomer.com/api/ai/list-evaluation-test-cases) |
| Create, read, update or delete a test case | [`POST /v1/ai/evaluation-testcases`](https://developer.kustomer.com/api/ai/create-an-evaluation-test-case), [`GET`](https://developer.kustomer.com/api/ai/get-evaluation-test-case-by-id), [`PUT`](https://developer.kustomer.com/api/ai/update-an-evaluation-test-case), [`DELETE /v1/ai/evaluation-testcases/{testCaseId}`](https://developer.kustomer.com/api/ai/delete-an-evaluation-test-case) |
| Start a run | [`POST /v1/ai/evaluations/{evaluationId}/run`](https://developer.kustomer.com/api/ai/run-an-evaluation) |
| Check a run's status and score | [`GET /v1/ai/evaluation-runs/{runId}`](https://developer.kustomer.com/api/ai/get-evaluation-run-by-id), or [`GET /v1/ai/evaluation-runs`](https://developer.kustomer.com/api/ai/list-evaluation-runs) with `evaluationIds` for history |
| See each result and why it failed | [`GET /v1/ai/evaluation-testcase-results`](https://developer.kustomer.com/api/ai/list-evaluation-test-case-results) with `evaluationRunIds`, or [`GET /v1/ai/evaluation-testcase-results/{resultId}`](https://developer.kustomer.com/api/ai/get-test-case-result-by-id) |

Starting a run fails with `404` if the evaluation doesn't exist or has no test cases, and with `409` if a run of that evaluation is already in progress, so poll the current run until it's `completed` before starting another. When you update a test case with `PUT`, send `input`, `resourceId` and `resourceType` every time; leaving out `toolCalls` keeps the existing ones, and `[]` clears them. All lists are paginated with `page` and `pageSize`, and every endpoint needs the AI Agents feature on your plan. Reading only needs read access to AI agents, but creating, changing and running evaluations requires an admin role.

Related: [AI and automations API](https://developer.kustomer.com/docs/ai/ai-api#evaluations), [AI automations](https://developer.kustomer.com/api/ai#automations), [Customers](https://developer.kustomer.com/api/core-resources#customers)

### Recipes

#### 1. Build an evaluation from a file of scenarios

Create an evaluation for an AI agent and load its test cases from a file you keep in version control, so the suite can be rebuilt or reviewed like code.

**Step 1.** [`POST /v1/ai/evaluations`](https://developer.kustomer.com/api/ai/create-an-evaluation.md). Create the evaluation with the agent's `automationId` (see [AI automations](https://developer.kustomer.com/api/ai#automations)) and a `name`. Names are unique per agent: a `409 Conflict` means one already exists, so find its ID with [`GET /v1/ai/automations/{automationId}/evaluations`](https://developer.kustomer.com/api/ai/list-evaluations-for-an-automation) instead. An agent can have up to 30 evaluations. Keep `data.id`.

```bash title="cURL"
curl --request POST "https://api.kustomerapp.com/v1/ai/evaluations" \
  --header "Authorization: Bearer $KUSTOMER_API_KEY" \
  --header "Content-Type: application/json" \
  --data "{\"automationId\": \"$AUTOMATION_ID\", \"name\": \"Refunds\"}"
```

```bash title="kusty"
kusty api POST /v1/ai/evaluations \
  --field automationId=$AUTOMATION_ID --field name=Refunds --fields id
```

**Step 2.** [`POST /v1/ai/evaluation-testcases`](https://developer.kustomer.com/api/ai/create-an-evaluation-test-case.md). Create one test case per scenario. Each needs a `name` (unique in the evaluation), the customer's `input`, and `resourceType: "customer"` with the `resourceId` of a customer whose data the agent's tools will see. Give every case an `expectedOutput` rubric, `toolCalls`, or both; a case with neither always passes. An evaluation holds up to 30 test cases, and creating more fails with `403`.

```bash title="cURL"
curl --request POST "https://api.kustomerapp.com/v1/ai/evaluation-testcases" \
  --header "Authorization: Bearer $KUSTOMER_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "evaluationId": "'"$EVALUATION_ID"'",
    "name": "Refund within policy",
    "input": "I would like a refund for order 4417, it arrived broken.",
    "resourceType": "customer",
    "resourceId": "'"$CUSTOMER_ID"'",
    "expectedOutput": "Apologizes and offers a full refund."
  }'
```

```bash title="kusty"
# cases.jsonl: one {"name", "input", "customerId", "expectedOutput"} per line
jq -c --arg ev "$EVALUATION_ID" \
  '{evaluationId: $ev, name, input, resourceType: "customer", resourceId: .customerId, expectedOutput}' cases.jsonl \
  | while read -r body; do
      kusty api POST /v1/ai/evaluation-testcases --data "$body" --fields id,attributes.name
    done
```

> `toolCalls` entries are matched to the tools the agent called by `id`, not by name, and an entry without an `id` is ignored.

#### 2. Turn a mishandled conversation into a regression test

When an AI agent gets a real conversation wrong, capture that moment as a test case so the fix can be verified and the mistake can't quietly come back.

**Step 1.** [`GET /v1/conversations/{id}/messages`](https://developer.kustomer.com/api/core-resources/get-messages-by-conversation.md). List the conversation's messages oldest first and find the customer message (`direction: in`) that the agent answered badly. Keep its `id`, the customer's ID from `relationships.customer`, and what the customer wrote.

```bash title="cURL"
curl "https://api.kustomerapp.com/v1/conversations/$CONVERSATION_ID/messages?sort=asc&pageSize=100" \
  --header "Authorization: Bearer $KUSTOMER_API_KEY"
```

```bash title="kusty"
kusty api GET /v1/conversations/$CONVERSATION_ID/messages --query sort=asc --all \
  | jq -r 'select(.data.attributes.direction == "in") | [.data.id, .data.relationships.customer.data.id, .data.attributes.preview] | @tsv'
```

**Step 2.** [`POST /v1/ai/evaluation-testcases`](https://developer.kustomer.com/api/ai/create-an-evaluation-test-case.md). Create the case with `conversationId` and `messageId` together. The conversation up to that message becomes the agent's context, and `input` is the message itself, so set `input` to what the customer wrote there. Describe the right answer in `expectedOutput`.

Check `multiTurnData` in the response: if the message isn't in that conversation, the case is still created, but without the conversation as context. The two source fields can't be changed later; create a new case instead.

```bash title="cURL"
curl --request POST "https://api.kustomerapp.com/v1/ai/evaluation-testcases" \
  --header "Authorization: Bearer $KUSTOMER_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "evaluationId": "'"$EVALUATION_ID"'",
    "name": "Escalates damaged-item refund",
    "conversationId": "'"$CONVERSATION_ID"'",
    "messageId": "'"$MESSAGE_ID"'",
    "input": "It arrived broken and I want my money back",
    "resourceType": "customer",
    "resourceId": "'"$CUSTOMER_ID"'",
    "expectedOutput": "Apologizes and offers a refund or replacement without asking for photos."
  }'
```

```bash title="kusty"
kusty api POST /v1/ai/evaluation-testcases \
  --field evaluationId=$EVALUATION_ID --field "name=Escalates damaged-item refund" \
  --field conversationId=$CONVERSATION_ID --field messageId=$MESSAGE_ID \
  --field "input=It arrived broken and I want my money back" \
  --field resourceType=customer --field resourceId=$CUSTOMER_ID \
  --field "expectedOutput=Apologizes and offers a refund or replacement without asking for photos." \
  --fields id,attributes.multiTurnData.sourceEndMessageId
```

#### 3. Gate a deploy on an agent's evaluation score

Run an evaluation from CI after changing an AI agent's draft, wait for the score, and fail the build with the judge's reasons when it drops below your bar.

**Step 1.** [`POST /v1/ai/evaluations/{evaluationId}/run`](https://developer.kustomer.com/api/ai/run-an-evaluation.md). Start a run. `runsPerTestCase` (up to 25) repeats each case to expose inconsistent answers; leave it out and each case runs 10 times. A `409` means a run of this evaluation is already pending or running, and a `404` means the evaluation doesn't exist or has no test cases. Keep `data.id`, the run ID.

```bash title="cURL"
curl --request POST "https://api.kustomerapp.com/v1/ai/evaluations/$EVALUATION_ID/run" \
  --header "Authorization: Bearer $KUSTOMER_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"runsPerTestCase": 3}'
```

```bash title="kusty"
RUN_ID=$(kusty api POST /v1/ai/evaluations/$EVALUATION_ID/run \
  --field runsPerTestCase:=3 | jq -r .data.id)
```

**Step 2.** [`GET /v1/ai/evaluation-runs/{runId}`](https://developer.kustomer.com/api/ai/get-evaluation-run-by-id.md). Poll the run every 10 to 30 seconds until `status` is `completed`, then compare `score` (0 to 100) with your threshold. `scoreByTestCase` has each case's percentage, keyed by test case ID.

```bash title="cURL"
curl "https://api.kustomerapp.com/v1/ai/evaluation-runs/$RUN_ID" \
  --header "Authorization: Bearer $KUSTOMER_API_KEY"
```

```bash title="kusty"
until [ "$(kusty api GET /v1/ai/evaluation-runs/$RUN_ID | jq -r .data.attributes.status)" = completed ]; do sleep 15; done
SCORE=$(kusty api GET /v1/ai/evaluation-runs/$RUN_ID | jq -r '.data.attributes.score // 0')
```

**Step 3.** [`GET /v1/ai/evaluation-testcase-results`](https://developer.kustomer.com/api/ai/list-evaluation-test-case-results.md). List the run's results and print the failures: those with `status: "error"` (with `error`) or `assertionResult.assertionFailed: true`, where `assertionResult.inputResult.explanation` gives the judge's reasoning. Then exit non-zero if the score was too low.

```bash title="cURL"
curl "https://api.kustomerapp.com/v1/ai/evaluation-testcase-results?evaluationRunIds=$RUN_ID&pageSize=250" \
  --header "Authorization: Bearer $KUSTOMER_API_KEY"
```

```bash title="kusty"
kusty api GET /v1/ai/evaluation-testcase-results --query evaluationRunIds=$RUN_ID --all \
  | jq -r '.data.attributes | select(.status == "error" or .assertionResult.assertionFailed)
      | [.testCaseId, .status, (.assertionResult.inputResult.explanation // .error)] | @tsv'
awk -v s="$SCORE" -v min=85 'BEGIN { exit (s < min) }'
```

> Kustomer keeps only the latest five runs of each evaluation. Starting a run deletes older runs and their results, so save any results you want to keep before the next run. Runs test the agent's current draft, so you can gate a change before turning it on.

#### 4. Update a test case's expected answer

Change what a good answer looks like for one test case, for example after a policy change, without touching its input, customer or expected tools.

**Step 1.** [`GET /v1/ai/evaluation-testcases/{testCaseId}`](https://developer.kustomer.com/api/ai/get-evaluation-test-case-by-id.md). Read the test case. `PUT` requires `input`, `resourceId` and `resourceType` every time, so copy them from here.

```bash title="cURL"
curl "https://api.kustomerapp.com/v1/ai/evaluation-testcases/$TEST_CASE_ID" \
  --header "Authorization: Bearer $KUSTOMER_API_KEY"
```

```bash title="kusty"
kusty api GET /v1/ai/evaluation-testcases/$TEST_CASE_ID > testcase.json
```

**Step 2.** [`PUT /v1/ai/evaluation-testcases/{testCaseId}`](https://developer.kustomer.com/api/ai/update-an-evaluation-test-case.md). Send the three required fields with the new `expectedOutput`. Fields you leave out, such as `name`, `resumeNode` and `toolCalls`, keep their current values; send `toolCalls: []` only if you mean to clear the expected tools. Renaming to a name already used in the evaluation fails with `409`.

```bash title="cURL"
curl --request PUT "https://api.kustomerapp.com/v1/ai/evaluation-testcases/$TEST_CASE_ID" \
  --header "Authorization: Bearer $KUSTOMER_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "input": "I would like a refund for order 4417, it arrived broken.",
    "resourceType": "customer",
    "resourceId": "'"$CUSTOMER_ID"'",
    "expectedOutput": "Apologizes and offers store credit or a replacement, not a cash refund."
  }'
```

```bash title="kusty"
jq -c --arg exp "Apologizes and offers store credit or a replacement, not a cash refund." \
  '.data.attributes | {input, resourceId, resourceType, expectedOutput: $exp}' testcase.json \
  | kusty api PUT /v1/ai/evaluation-testcases/$TEST_CASE_ID --data @- --fields attributes.expectedOutput
```

### Example object

A real record from [`GET /v1/ai/evaluations/{evaluationId}`](https://developer.kustomer.com/api/ai/get-evaluation-by-id.md):

```json
{
  "id": "507f1f77bcf86cd799439011",
  "type": "evaluation",
  "attributes": {
    "orgId": "507f1f77bcf86cd799439010",
    "name": "Q1 2024 Evaluation",
    "automationId": "507f1f77bcf86cd799439012",
    "createdAt": "2024-01-15T10:30:00.000Z",
    "createdBy": "507f1f77bcf86cd799439099",
    "updatedAt": "2024-01-15T10:30:00.000Z"
  },
  "relationships": {
    "org": {
      "links": {
        "self": "/v1/orgs/507f1f77bcf86cd799439010"
      },
      "data": {
        "type": "org",
        "id": "507f1f77bcf86cd799439010"
      }
    }
  },
  "links": {
    "self": "string"
  }
}
```

Fields:

- `id` (string): The `evaluationId` you send when creating test cases, starting a run, and filtering test cases or runs with `evaluationIds`.
- `type` (string): Resource type
- `attributes` (object): Evaluation attributes
- `attributes.orgId` (string): Organization ID
- `attributes.name` (string): Unique among one automation's evaluations; a duplicate fails with `409 Conflict`. Spaces at either end are trimmed, and an evaluation created without a name takes the automation's name. The only field `PUT` changes.
- `attributes.automationId` (string): The [AI automation](https://developer.kustomer.com/api/ai#automations) the evaluation tests. Set on create and can't be changed. An automation can have up to 30 evaluations; creating more fails with `403`.
- `attributes.createdAt` (string (date-time)): Creation timestamp
- `attributes.createdBy` (string): The ID of the user who created the evaluation. Runs, test cases and results record `createdBy` the same way.
- `attributes.updatedAt` (string (date-time)): Last update timestamp
- `links.self` (string): URL for this resource

### Relationships

- Points to **AI automations** via `relationships.automation`: The AI automation an evaluation tests, from the evaluation's `automationId`. An automation can have up to 30 evaluations, listed with `GET /v1/ai/automations/{automationId}/evaluations`.

### Permissions

Fewest roles for a read-only integration: `org.permission.assistant.read`. For every endpoint: `org.permission.assistant.read`, `org.permission.assistant.create`, `org.permission.assistant.delete`, `org.permission.assistant.update`.

## Path parameters

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `evaluationId` | string | yes | Evaluation ID |

## Request body

Content type: `application/json` (required)

- `name` (string): Name of the evaluation (max length 100)

Example:

```json
{
  "name": "Q1 2024 Evaluation Updated"
}
```

## Responses

### 200

OK

- `data` (object)
  - `id` (string): Unique evaluation identifier
  - `type` (string): Resource type
  - `attributes` (object): Evaluation attributes
    - `orgId` (string): Organization ID
    - `name` (string): Name of the evaluation
    - `automationId` (string): Associated automation ID
    - `createdAt` (string<date-time>): Creation timestamp
    - `createdBy` (string): User ID that created the evaluation
    - `updatedAt` (string<date-time>): Last update timestamp
  - `relationships` (object)
    - `org` (object)
  - `links` (object)
    - `self` (string): URL for this resource

```json
{
  "data": {
    "id": "507f1f77bcf86cd799439011",
    "type": "evaluation",
    "attributes": {
      "orgId": "507f1f77bcf86cd799439010",
      "name": "Q1 2024 Evaluation",
      "automationId": "507f1f77bcf86cd799439012",
      "createdAt": "2024-01-15T10:30:00.000Z",
      "createdBy": "507f1f77bcf86cd799439099",
      "updatedAt": "2024-01-15T10:30:00.000Z"
    },
    "relationships": {
      "org": {
        "links": {
          "self": "/v1/orgs/507f1f77bcf86cd799439010"
        },
        "data": {
          "type": "org",
          "id": "507f1f77bcf86cd799439010"
        }
      }
    },
    "links": {
      "self": "string"
    }
  }
}
```

### 400

Bad Request - Invalid request parameters or body

### 401

Unauthorized - Authentication required or invalid token

### 403

Forbidden - Insufficient permissions or AI Agents feature not enabled

### 404

Not Found - Resource does not exist

### 409

Conflict - Request conflicts with the current resource state

## Example request

```bash
curl --request PUT \
  --url 'https://api.kustomerapp.com/v1/ai/evaluations/507f1f77bcf86cd799439011' \
  --header "Authorization: Bearer $KUSTOMER_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
  "name": "Q1 2024 Evaluation Updated"
}'
```