Moderate an AI agent

An AI agent reads content it did not write and produces content nobody has read yet. This guide shows where the two checks go, with the module of the Node.js guide. The calls are the same from any language.

What to check

One image or one text is one check, whatever the number of categories it is checked against. A run that checks one input and one output is two checks.

The two calls

Check the input, run the model, check the output. A check that cannot be made holds the content: it is never published unchecked.

import { checkText, ModerationError } from "./moderation"

type Verdict = "accept" | "review" | "reject"

// A check that cannot be made is a "review": hold the content.
async function verdict(text: string, ref: string): Promise<Verdict> {
  try {
    return (await checkText(text, ref, "support_agent")).action
  } catch (error) {
    // The content cannot be accepted as it is: refuse it.
    if (error instanceof ModerationError && [400, 413, 415].includes(error.status)) return "reject"
    // Anything else, a timeout included: hold it.
    return "review"
  }
}

// model, refuse and holdForReview are yours.
export async function answer(run: { id: string }, message: string) {
  // Before the model: what the user sent.
  if ((await verdict(message, `${run.id}:in`)) === "reject") return refuse(run)

  const reply = await model.generate(message)

  // Before you publish: what the agent wrote.
  if ((await verdict(reply, `${run.id}:out`)) !== "accept") return holdForReview(run, reply)
  return reply
}

support_agent is a workflow you created in the dashboard. Leave the third argument out to use default.

For an image the agent received or generated, call checkImage the same way: the third argument is the workflow.

One ref per step

ref is your own id for the content, 1 to 128 characters with no control character. Use the run id and the step, such as run_8f3a:in and run_8f3a:out. It comes back in the answer and stays in the journal, so when a reply was held you filter the journal by the start of the ref (run_8f3a) and see the decision, the reasons with their scores and thresholds, and the workflow version. The content is never kept.

A workflow per agent

A workflow decides which categories are checked and how strictly. Give a customer-facing agent a strict one and an internal tool a lenient one. Create one key for each agent, limit it to the workflow it needs and to images, texts or both: a call outside that gets 403 forbidden. Authentication and keys explains scopes.

When a check cannot be made

Responses and errors lists every type.

Long outputs

A text is at most 20,000 characters (5,000 on the free plan). A longer output gets 413 text_too_long. Split it at paragraph boundaries and check each part: each part is one check.

Keep the key on your servers

The agent's runtime calls the API with a key. If the agent runs in a browser or on a user's device, it must call your own backend, which calls openshield: the API refuses requests that carry a browser Origin header. Best practices says more.

What it does not do

• It checks images and text. It does not check video or audio.
• It checks content against categories such as unsafe text, hate and personal data. It does not detect prompt injection.
• Nobody reviews content for you. A review decision goes back to your own team, and the content was never stored at openshield.