---
title: "Moderate an AI agent"
description: "Check what an agent reads and what it writes with two calls and a fallback, so nothing unchecked reaches the model or the reader."
canonical_url: "https://openshield.dev/docs/integration-guides/moderate-ai-agents"
---
# Moderate an AI agent

> Check what an agent reads and what it writes with two calls and a fallback, so nothing unchecked reaches the model or the reader.

An AI agent reads content it did not write and produces content nobody has read yet. This guide shows where the two checks go, with the module of the [Node.js guide](https://openshield.dev/docs/integration-guides/nodejs). The calls are the same from any language.

## What to check

<table>
<thead>
  <tr>
    <th>
      Side
    </th>
    
    <th>
      What it is
    </th>
    
    <th>
      When to check
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Input
    </td>
    
    <td>
      The messages and images a user sends the agent.
    </td>
    
    <td>
      Before they reach the model or a tool.
    </td>
  </tr>
  
  <tr>
    <td>
      Output
    </td>
    
    <td>
      The text and images the agent generates, and the tool results it reposts.
    </td>
    
    <td>
      Before you publish or send them.
    </td>
  </tr>
</tbody>
</table>

One image or one text is one check, whatever the number of categories it is checked against. A run that checks one input and one output is two checks.

## The two calls

Check the input, run the model, check the output. A check that cannot be made holds the content: it is never published unchecked.

```ts [agent.ts]
import { checkText, ModerationError } from "./moderation"

type Verdict = "accept" | "review" | "reject"

// A check that cannot be made is a "review": hold the content.
async function verdict(text: string, ref: string): Promise<Verdict> {
  try {
    return (await checkText(text, ref, "support_agent")).action
  } catch (error) {
    // The content cannot be accepted as it is: refuse it.
    if (error instanceof ModerationError && [400, 413, 415].includes(error.status)) return "reject"
    // Anything else, a timeout included: hold it.
    return "review"
  }
}

// model, refuse and holdForReview are yours.
export async function answer(run: { id: string }, message: string) {
  // Before the model: what the user sent.
  if ((await verdict(message, `${run.id}:in`)) === "reject") return refuse(run)

  const reply = await model.generate(message)

  // Before you publish: what the agent wrote.
  if ((await verdict(reply, `${run.id}:out`)) !== "accept") return holdForReview(run, reply)
  return reply
}
```

`support_agent` is a workflow you created in the [dashboard](https://openshield.dev/dashboard/workflows). Leave the third argument out to use `default`.

For an image the agent received or generated, call `checkImage` the same way: the third argument is the workflow.

## One ref per step

`ref` is your own id for the content, 1 to 128 characters with no control character. Use the run id and the step, such as `run_8f3a:in` and `run_8f3a:out`. It comes back in the answer and stays in the [journal](https://openshield.dev/docs/journal-and-ref), so when a reply was held you filter the journal by the start of the ref (`run_8f3a`) and see the decision, the reasons with their scores and thresholds, and the workflow version. The content is never kept.

<warning>

Do not put personal data in a `ref`. Use an opaque id from your own database.

</warning>

## A workflow per agent

A [workflow](https://openshield.dev/docs/workflows) decides which categories are checked and how strictly. Give a customer-facing agent a strict one and an internal tool a lenient one. Create one key for each agent, limit it to the workflow it needs and to images, texts or both: a call outside that gets `403 forbidden`. [Authentication and keys](https://openshield.dev/docs/authentication) explains scopes.

## When a check cannot be made

<table>
<thead>
  <tr>
    <th>
      Answer
    </th>
    
    <th>
      What to do
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        400
      </code>
      
      , <code>
        413
      </code>
      
      , <code>
        415
      </code>
    </td>
    
    <td>
      The content cannot be accepted as it is: refuse it. For <code>
        413 text_too_long
      </code>
      
      , see below.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        401
      </code>
      
      , <code>
        403
      </code>
    </td>
    
    <td>
      A configuration problem. Log it, without the key, and alert your team.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        429
      </code>
      
      , <code>
        503
      </code>
      
       with <code>
        Retry-After
      </code>
    </td>
    
    <td>
      Wait that many seconds and retry, at most twice.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        500
      </code>
      
      , a timeout, a network error
    </td>
    
    <td>
      Fail closed: hold the content.
    </td>
  </tr>
</tbody>
</table>

[Responses and errors](https://openshield.dev/docs/responses-and-errors) lists every type.

## Long outputs

A text is at most 20,000 characters (5,000 on the free plan). A longer output gets `413 text_too_long`. Split it at paragraph boundaries and check each part: each part is one check.

## Keep the key on your servers

The agent's runtime calls the API with a key. If the agent runs in a browser or on a user's device, it must call your own backend, which calls openshield: the API refuses requests that carry a browser `Origin` header. [Best practices](https://openshield.dev/docs/best-practices) says more.

## What it does not do

- It checks images and text. It does not check video or audio.
- It checks content against categories such as unsafe text, hate and personal data. It does not detect prompt injection.
- Nobody reviews content for you. A `review` decision goes back to your own team, and the content was never stored at openshield.


## Sitemap

See the full [sitemap](https://openshield.dev/sitemap.md) for all pages.