# AI and automations API

> Read AI-generated summaries of conversations, customers and custom objects, list your AI agents, and test them with evaluations, test cases and runs. Includes a CI script that fails a build when an agent's score drops.

Source: https://developer.kustomer.com/docs/ai/ai-api

The AI API covers three things:

| Area                        | Endpoints                                                                                                           | Use it to                                                                               |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| [Summaries](#summaries)     | `/v1/ai/summaries`                                                                                                  | Read the AI summary of a conversation, customer or custom object, optionally translated |
| [Automations](#automations) | `/v1/ai/automations`                                                                                                | List your AI agents and look one up. Read-only: agents are built in Kustomer.           |
| [Evaluations](#evaluations) | `/v1/ai/evaluations`, `/v1/ai/evaluation-testcases`, `/v1/ai/evaluation-runs`, `/v1/ai/evaluation-testcase-results` | Test an AI agent against a suite of scenarios and score it, from a script or CI         |

All endpoints use the same [authentication](https://developer.kustomer.com/docs/get-started/authentication) and [JSON:API format](https://developer.kustomer.com/docs/concepts/json-api-format) as the rest of the API, under `https://<org>.api.kustomerapp.com`. Every operation is in the [API reference](https://developer.kustomer.com/api/ai).

> **Plan requirements:** The evaluation and automation endpoints need the **AI Agents** feature. Summaries need **AI Copilot** or **AI Summaries**. Without the feature, requests return `403 Forbidden`.

## Summaries

Kustomer generates summaries automatically and updates them as a conversation, customer or custom object changes. The API reads them; it doesn't create them.

`GET /v1/ai/summaries/{resourceId}` ([reference](https://developer.kustomer.com/api/ai/get-ai-summary-by-resource-id))

```bash title="cURL"
curl "https://acme.api.kustomerapp.com/v1/ai/summaries/6716a3a9c4e5f60012ab34c1?lang=fr" \
  -H "Authorization: Bearer $KUSTOMER_API_KEY"
```

```json title="Response"
{
  "data": {
    "type": "summary",
    "id": "6716a4f0c4e5f60012ab34f2",
    "attributes": {
      "resourceId": "6716a3a9c4e5f60012ab34c1",
      "resourceType": "conversation",
      "text": "Le client a demandé où en était sa commande 4417 et a demandé un remboursement…",
      "createdAt": "2026-10-14T15:02:11.000Z",
      "updatedAt": "2026-10-14T15:20:43.000Z",
      "lastSummarizedAt": "2026-10-14T15:20:43.000Z"
    }
  }
}
```

- `resourceType` is `conversation`, `customer` or `kobject`.
- `lang` translates the text (for example `fr`, `es`, `de`). If translation fails, you get the summary in its original language.
- `lastSummarizedAt` tells you how fresh the summary is.

To fetch many at once, pass up to 100 IDs:

`GET /v1/ai/summaries/batch` ([reference](https://developer.kustomer.com/api/ai/batch-get-ai-summaries-by-resource-ids))

```bash title="cURL"
curl -G "https://acme.api.kustomerapp.com/v1/ai/summaries/batch" \
  -H "Authorization: Bearer $KUSTOMER_API_KEY" \
  --data-urlencode "resourceIds=6716a3a9c4e5f60012ab34c1,6716a3b2c4e5f60012ab34c7"
```

The response's `data` is an array. Resources without a summary are left out rather than returned as errors, so match results on `attributes.resourceId`. IDs must be 24-character hex strings. `page`, `pageSize` and `lang` work as above.

> **Summaries in your own tools:** A common pattern: subscribe to `kustomer.conversation.update` [webhooks](https://developer.kustomer.com/docs/events/outbound-webhooks), batch the conversation IDs, and fetch their summaries for your CRM, data warehouse or Slack digests.

## Automations

Automations are the AI agents you build in Kustomer. The API lists them so you can find the ID to evaluate; creating and editing agents happens in Kustomer.

`GET /v1/ai/automations` ([reference](https://developer.kustomer.com/api/ai/list-ai-automations))

| Query parameter    | Meaning                                                               |
| ------------------ | --------------------------------------------------------------------- |
| `type`             | `customer` (default), `rep`, `conversational` or `workflow`           |
| `name`             | Filter by name                                                        |
| `triggers`         | Comma-separated trigger event names                                   |
| `page`, `pageSize` | [Pagination](https://developer.kustomer.com/docs/concepts/pagination) |

Each automation has `name`, `description`, `type`, `machineUserId` (the user the agent acts as), `version`, and timestamps. The API returns version 0, the agent's current draft, which is also what evaluations test. Get one with `GET /v1/ai/automations/{automationId}`.

## Evaluations

An evaluation is a test suite for one AI agent. You write test cases (a customer message plus what a good answer looks like), run them, and get a score.

```mermaid
flowchart LR
  A["Automation<br/>(AI agent)"] --> E["Evaluation<br/>a named suite"]
  E --> T1["Test case<br/>input · expectedOutput · toolCalls"]
  E --> T2["Test case"]
  E --> R["Run<br/>status · score · scoreByTestCase"]
  R --> X1["Result<br/>output · assertionResult · traceId"]
  R --> X2["Result"]
```

_An evaluation run for a refunds agent with four test cases, each run three times. Results appear as pass or fail dots: tool not called, expected answer not met, or engine error count as failures. Selecting a result shows the agent's output and the judge's explanation and confidence. Each test case's score is the share of its results that passed, and the run score is the overall percentage._

### How scoring works

Each run executes every test case `runsPerTestCase` times against the agent. An LLM judge scores each result **pass or fail**:

| A result fails when                      | Shown in                                                                                  |
| ---------------------------------------- | ----------------------------------------------------------------------------------------- |
| The agent errored                        | `status: "error"` and `error`                                                             |
| The answer doesn't meet `expectedOutput` | `assertionResult.inputResult.met: false`, with the judge's `explanation` and `confidence` |
| An expected tool wasn't called           | `assertionResult.toolCallsResult[].toolCalled: false`                                     |

A test case's score is the percentage of its results that passed (`scoreByTestCase`), and the run's `score` is the overall percentage, from 0 to 100. A test case with neither `expectedOutput` nor `toolCalls` always passes, so give every case at least one.

During a run, tools with side effects (sending messages, changing data, non-GET HTTP calls) aren't executed, but the call is still recorded, so you can assert that the agent _would_ have called `IssueRefund` without issuing one.

### 1. Create an evaluation

`POST /v1/ai/evaluations` ([reference](https://developer.kustomer.com/api/ai/create-an-evaluation))

```bash title="cURL"
curl -X POST "https://acme.api.kustomerapp.com/v1/ai/evaluations" \
  -H "Authorization: Bearer $KUSTOMER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "automationId": "6716c0a1c4e5f60012ab36d0", "name": "Refunds" }'
```

List an agent's evaluations with `GET /v1/ai/automations/{automationId}/evaluations`.

### 2. Add test cases

`POST /v1/ai/evaluation-testcases` ([reference](https://developer.kustomer.com/api/ai/create-an-evaluation-test-case))

```json title="Request body"
{
  "evaluationId": "6716c1d2c4e5f60012ab36ea",
  "name": "Refund within policy",
  "input": "I'd like a refund for order 4417, it arrived broken.",
  "resourceType": "customer",
  "resourceId": "6716a3a9c4e5f60012ab34c1",
  "expectedOutput": "Apologizes, confirms the order is within the 30-day window, and offers a full refund.",
  "toolCalls": [{ "id": "6716c2e3c4e5f60012ab3701", "name": "IssueRefund" }]
}
```

**Test case**

Turn real conversations into regression tests: when an agent mishandles a chat, create a test case with that conversation's `conversationId`, the customer's `messageId`, and an `expectedOutput` describing the right answer.

Update a test case with `PUT /v1/ai/evaluation-testcases/{testCaseId}`. Send `input`, `resourceId` and `resourceType` each time. Leaving out `toolCalls` keeps the existing ones, and `[]` clears them. The conversation fields can't be changed after creation.

### 3. Run it

`POST /v1/ai/evaluations/{evaluationId}/run` ([reference](https://developer.kustomer.com/api/ai/run-an-evaluation))

```bash title="cURL"
curl -X POST "https://acme.api.kustomerapp.com/v1/ai/evaluations/6716c1d2c4e5f60012ab36ea/run" \
  -H "Authorization: Bearer $KUSTOMER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "runsPerTestCase": 3 }'
```

The response is a run with `status: "pending"`. It moves to `running` and then `completed`.

| Status | Meaning                                           |
| ------ | ------------------------------------------------- |
| `404`  | The evaluation doesn't exist or has no test cases |
| `409`  | A run of this evaluation is already in progress   |

Running the same case several times exposes flaky behavior. Each run uses AI capacity in proportion to test cases × `runsPerTestCase`, so size it deliberately.

### 4. Read the results

Poll the run until it completes:

`GET /v1/ai/evaluation-runs/{runId}` ([reference](https://developer.kustomer.com/api/ai/get-evaluation-run-by-id))

```json title="Completed run"
{
  "data": {
    "type": "evaluation-run",
    "id": "6716c3f4c4e5f60012ab3712",
    "attributes": {
      "evaluationId": "6716c1d2c4e5f60012ab36ea",
      "status": "completed",
      "score": 83.3,
      "scoreByTestCase": { "6716c2a0c4e5f60012ab36f5": 100, "6716c2b1c4e5f60012ab36f9": 66.7 },
      "runsPerTestCase": 3,
      "durationInNanos": 41200000000
    }
  }
}
```

Then list the individual results to see what failed and why:

`GET /v1/ai/evaluation-testcase-results` ([reference](https://developer.kustomer.com/api/ai/list-evaluation-test-case-results))

```bash title="Failed results with the judge's reasoning"
curl -sG "https://acme.api.kustomerapp.com/v1/ai/evaluation-testcase-results" \
  -H "Authorization: Bearer $KUSTOMER_API_KEY" \
  --data-urlencode "evaluationRunIds=6716c3f4c4e5f60012ab3712" \
  --data-urlencode "pageSize=100" \
| jq -r '.data[].attributes
    | select(.status == "error" or .assertionResult.assertionFailed)
    | [.testCaseId, .status, (.assertionResult.inputResult.explanation // .error)] | @tsv'
```

Each result has the agent's `output`, the tools it called (`toolCalls[].canonicalName`, `toolInput`, `toolOutput`), `assertionResult`, timing, and a `traceId` for following the run step by step in Kustomer.

## Gate deploys on agent quality

Because runs test the agent's draft, you can check a change before turning it on. This script runs an evaluation, waits for it, and exits non-zero if the score is below a threshold, ready for a CI job:

```ts title="eval-gate.ts"
const BASE = `https://${process.env.KUSTOMER_ORG}.api.kustomerapp.com`
const EVALUATION_ID = process.env.EVALUATION_ID!
const MIN_SCORE = Number(process.env.MIN_SCORE ?? 85)

async function api<T>(method: string, path: string, body?: unknown): Promise<T> {
  const res = await fetch(BASE + path, {
    method,
    headers: { Authorization: `Bearer ${process.env.KUSTOMER_API_KEY}`, 'Content-Type': 'application/json' },
    body: body ? JSON.stringify(body) : undefined,
  })
  if (!res.ok) throw new Error(`${method} ${path}: ${res.status} ${await res.text()}`)
  return res.json() as Promise<T>
}

type Run = { data: { id: string; attributes: { status: 'pending' | 'running' | 'completed'; score?: number } } }

const started = await api<Run>('POST', `/v1/ai/evaluations/${EVALUATION_ID}/run`, { runsPerTestCase: 3 })
let run = started
while (run.data.attributes.status !== 'completed') {
  await new Promise((r) => setTimeout(r, 10_000))
  run = await api<Run>('GET', `/v1/ai/evaluation-runs/${started.data.id}`)
}

const score = run.data.attributes.score ?? 0
console.log(`Evaluation score: ${score} (minimum ${MIN_SCORE})`)
if (score < MIN_SCORE) process.exit(1)
```

With [kusty](https://developer.kustomer.com/docs/ai/kusty), the same steps are `kusty evaluations runevaluation --evaluationId … --data '{"runsPerTestCase":3}'` and `kusty evaluation-runs get --runId …`, or ask your AI assistant through the [kusty MCP server](https://developer.kustomer.com/docs/ai/kusty-mcp) to run the evals and explain the failures.

## Endpoint summary

| Operation                                 | Endpoint                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Get a summary                             | [`GET /v1/ai/summaries/{resourceId}`](https://developer.kustomer.com/api/ai/get-ai-summary-by-resource-id)                                                                                                                                                                                                                                                                                                                                                                     |
| Get summaries in bulk                     | [`GET /v1/ai/summaries/batch`](https://developer.kustomer.com/api/ai/batch-get-ai-summaries-by-resource-ids)                                                                                                                                                                                                                                                                                                                                                                   |
| List automations                          | [`GET /v1/ai/automations`](https://developer.kustomer.com/api/ai/list-ai-automations)                                                                                                                                                                                                                                                                                                                                                                                          |
| Get an automation                         | [`GET /v1/ai/automations/{automationId}`](https://developer.kustomer.com/api/ai/get-ai-automation-by-id)                                                                                                                                                                                                                                                                                                                                                                       |
| List an automation's evaluations          | [`GET /v1/ai/automations/{automationId}/evaluations`](https://developer.kustomer.com/api/ai/list-evaluations-for-an-automation)                                                                                                                                                                                                                                                                                                                                                |
| Create, get, update, delete an evaluation | [`POST /v1/ai/evaluations`](https://developer.kustomer.com/api/ai/create-an-evaluation), [`GET`](https://developer.kustomer.com/api/ai/get-evaluation-by-id), [`PUT`](https://developer.kustomer.com/api/ai/update-an-evaluation), [`DELETE /v1/ai/evaluations/{evaluationId}`](https://developer.kustomer.com/api/ai/delete-an-evaluation)                                                                                                                                    |
| Run an evaluation                         | [`POST /v1/ai/evaluations/{evaluationId}/run`](https://developer.kustomer.com/api/ai/run-an-evaluation)                                                                                                                                                                                                                                                                                                                                                                        |
| Test cases                                | [`GET`](https://developer.kustomer.com/api/ai/list-evaluation-test-cases), [`POST /v1/ai/evaluation-testcases`](https://developer.kustomer.com/api/ai/create-an-evaluation-test-case), [`GET`](https://developer.kustomer.com/api/ai/get-evaluation-test-case-by-id), [`PUT`](https://developer.kustomer.com/api/ai/update-an-evaluation-test-case), [`DELETE /v1/ai/evaluation-testcases/{testCaseId}`](https://developer.kustomer.com/api/ai/delete-an-evaluation-test-case) |
| Runs                                      | [`GET /v1/ai/evaluation-runs`](https://developer.kustomer.com/api/ai/list-evaluation-runs), [`GET /v1/ai/evaluation-runs/{runId}`](https://developer.kustomer.com/api/ai/get-evaluation-run-by-id)                                                                                                                                                                                                                                                                             |
| Results                                   | [`GET /v1/ai/evaluation-testcase-results`](https://developer.kustomer.com/api/ai/list-evaluation-test-case-results), [`GET /v1/ai/evaluation-testcase-results/{resultId}`](https://developer.kustomer.com/api/ai/get-test-case-result-by-id)                                                                                                                                                                                                                                   |