AI and automations API

The AI API covers three things:

All endpoints use the same authentication and JSON:API format as the rest of the API, under https://<org>.api.kustomerapp.com. Every operation is in the API reference.

Summaries

Kustomer generates summaries automatically and updates them as a conversation, customer or custom object changes. The API reads them; it doesn't create them.

curl "https://acme.api.kustomerapp.com/v1/ai/summaries/6716a3a9c4e5f60012ab34c1?lang=fr" \
  -H "Authorization: Bearer $KUSTOMER_API_KEY"

{
  "data": {
    "type": "summary",
    "id": "6716a4f0c4e5f60012ab34f2",
    "attributes": {
      "resourceId": "6716a3a9c4e5f60012ab34c1",
      "resourceType": "conversation",
      "text": "Le client a demandé où en était sa commande 4417 et a demandé un remboursement…",
      "createdAt": "2026-10-14T15:02:11.000Z",
      "updatedAt": "2026-10-14T15:20:43.000Z",
      "lastSummarizedAt": "2026-10-14T15:20:43.000Z"
    }
  }
}

• resourceType is conversation, customer or kobject.
• lang translates the text (for example fr, es, de). If translation fails, you get the summary in its original language.
• lastSummarizedAt tells you how fresh the summary is.

To fetch many at once, pass up to 100 IDs:

curl -G "https://acme.api.kustomerapp.com/v1/ai/summaries/batch" \
  -H "Authorization: Bearer $KUSTOMER_API_KEY" \
  --data-urlencode "resourceIds=6716a3a9c4e5f60012ab34c1,6716a3b2c4e5f60012ab34c7"

The response's data is an array. Resources without a summary are left out rather than returned as errors, so match results on attributes.resourceId. IDs must be 24-character hex strings. page, pageSize and lang work as above.

Automations

Automations are the AI agents you build in Kustomer. The API lists them so you can find the ID to evaluate; creating and editing agents happens in Kustomer.

Each automation has name, description, type, machineUserId (the user the agent acts as), version, and timestamps. The API returns version 0, the agent's current draft, which is also what evaluations test. Get one with GET /v1/ai/automations/{automationId}.

Evaluations

An evaluation is a test suite for one AI agent. You write test cases (a customer message plus what a good answer looks like), run them, and get a score.

flowchart LR
  A["Automation<br/>(AI agent)"] --> E["Evaluation<br/>a named suite"]
  E --> T1["Test case<br/>input · expectedOutput · toolCalls"]
  E --> T2["Test case"]
  E --> R["Run<br/>status · score · scoreByTestCase"]
  R --> X1["Result<br/>output · assertionResult · traceId"]
  R --> X2["Result"]

How scoring works

Each run executes every test case runsPerTestCase times against the agent. An LLM judge scores each result pass or fail:

A test case's score is the percentage of its results that passed (scoreByTestCase), and the run's score is the overall percentage, from 0 to 100. A test case with neither expectedOutput nor toolCalls always passes, so give every case at least one.

During a run, tools with side effects (sending messages, changing data, non-GET HTTP calls) aren't executed, but the call is still recorded, so you can assert that the agent would have called IssueRefund without issuing one.

1. Create an evaluation

curl -X POST "https://acme.api.kustomerapp.com/v1/ai/evaluations" \
  -H "Authorization: Bearer $KUSTOMER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "automationId": "6716c0a1c4e5f60012ab36d0", "name": "Refunds" }'

List an agent's evaluations with GET /v1/ai/automations/{automationId}/evaluations.

2. Add test cases

{
  "evaluationId": "6716c1d2c4e5f60012ab36ea",
  "name": "Refund within policy",
  "input": "I'd like a refund for order 4417, it arrived broken.",
  "resourceType": "customer",
  "resourceId": "6716a3a9c4e5f60012ab34c1",
  "expectedOutput": "Apologizes, confirms the order is within the 30-day window, and offers a full refund.",
  "toolCalls": [{ "id": "6716c2e3c4e5f60012ab3701", "name": "IssueRefund" }]
}

Turn real conversations into regression tests: when an agent mishandles a chat, create a test case with that conversation's conversationId, the customer's messageId, and an expectedOutput describing the right answer.

Update a test case with PUT /v1/ai/evaluation-testcases/{testCaseId}. Send input, resourceId and resourceType each time. Leaving out toolCalls keeps the existing ones, and [] clears them. The conversation fields can't be changed after creation.

3. Run it

curl -X POST "https://acme.api.kustomerapp.com/v1/ai/evaluations/6716c1d2c4e5f60012ab36ea/run" \
  -H "Authorization: Bearer $KUSTOMER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "runsPerTestCase": 3 }'

The response is a run with status: "pending". It moves to running and then completed.

Running the same case several times exposes flaky behavior. Each run uses AI capacity in proportion to test cases × runsPerTestCase, so size it deliberately.

4. Read the results

Poll the run until it completes:

{
  "data": {
    "type": "evaluation-run",
    "id": "6716c3f4c4e5f60012ab3712",
    "attributes": {
      "evaluationId": "6716c1d2c4e5f60012ab36ea",
      "status": "completed",
      "score": 83.3,
      "scoreByTestCase": { "6716c2a0c4e5f60012ab36f5": 100, "6716c2b1c4e5f60012ab36f9": 66.7 },
      "runsPerTestCase": 3,
      "durationInNanos": 41200000000
    }
  }
}

Then list the individual results to see what failed and why:

curl -sG "https://acme.api.kustomerapp.com/v1/ai/evaluation-testcase-results" \
  -H "Authorization: Bearer $KUSTOMER_API_KEY" \
  --data-urlencode "evaluationRunIds=6716c3f4c4e5f60012ab3712" \
  --data-urlencode "pageSize=100" \
| jq -r '.data[].attributes
    | select(.status == "error" or .assertionResult.assertionFailed)
    | [.testCaseId, .status, (.assertionResult.inputResult.explanation // .error)] | @tsv'

Each result has the agent's output, the tools it called (toolCalls[].canonicalName, toolInput, toolOutput), assertionResult, timing, and a traceId for following the run step by step in Kustomer.

Gate deploys on agent quality

Because runs test the agent's draft, you can check a change before turning it on. This script runs an evaluation, waits for it, and exits non-zero if the score is below a threshold, ready for a CI job:

const BASE = `https://${process.env.KUSTOMER_ORG}.api.kustomerapp.com`
const EVALUATION_ID = process.env.EVALUATION_ID!
const MIN_SCORE = Number(process.env.MIN_SCORE ?? 85)

async function api<T>(method: string, path: string, body?: unknown): Promise<T> {
  const res = await fetch(BASE + path, {
    method,
    headers: { Authorization: `Bearer ${process.env.KUSTOMER_API_KEY}`, 'Content-Type': 'application/json' },
    body: body ? JSON.stringify(body) : undefined,
  })
  if (!res.ok) throw new Error(`${method} ${path}: ${res.status} ${await res.text()}`)
  return res.json() as Promise<T>
}

type Run = { data: { id: string; attributes: { status: 'pending' | 'running' | 'completed'; score?: number } } }

const started = await api<Run>('POST', `/v1/ai/evaluations/${EVALUATION_ID}/run`, { runsPerTestCase: 3 })
let run = started
while (run.data.attributes.status !== 'completed') {
  await new Promise((r) => setTimeout(r, 10_000))
  run = await api<Run>('GET', `/v1/ai/evaluation-runs/${started.data.id}`)
}

const score = run.data.attributes.score ?? 0
console.log(`Evaluation score: ${score} (minimum ${MIN_SCORE})`)
if (score < MIN_SCORE) process.exit(1)

With kusty, the same steps are kusty evaluations runevaluation --evaluationId … --data '{"runsPerTestCase":3}' and kusty evaluation-runs get --runId …, or ask your AI assistant through the kusty MCP server to run the evals and explain the failures.

Endpoint summary