Bots and AI agents

Allow, challenge or block every AI bot.

Each request is classified as AI training, AI search, a user’s AI agent, a search engine, unverified automation or a spoofed crawler. You decide what each category, or each bot, may do on each site, and the crawlers that lie lose their free pass.

Create account See MCP Shield

OPENWAFBots & AI agents

Allow, challenge or block every AI bot, and trap the ones that lie

allow
Allow
Verified search and assistant bots get through, checked against their operators’ published ranges, reverse DNS or Web Bot Auth signatures.
challenge
Challenge
Unverified automation gets a proof-of-work page; real browsers pass, scripts that claim to be Googlebot don’t.
block
Block
Turn training crawlers away site by site, and publish your preferences in a managed robots.txt with Content-Signal.
tarpit
Trap and tarpit
A hidden link marks the readers that ignore the rules, and an optional tarpit keeps them busy instead of your origin.

Opt-in and off by default. Start in log mode, look at the numbers, then enforce.

Bots and AI agentsDecide what each AI crawler may do, and catch the fakes.

Per site and per bot. In beta: opt-in, off by default, and it starts in log mode.

# OpenWAF AI agents (managed)
User-agent: gptbot
Disallow: /

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /__waf/
# End OpenWAF AI agents

Verified crawlers
A catalogue of 31 bots from 18 operators, refreshed daily from 20 official sources and checked by IP range, reverse DNS and Web Bot Auth signatures. Allow search, block training, one switch per bot.
Spoofed crawlers get the challenge
A User-Agent that claims to be Googlebot from a cloud server is a fake: it no longer skips the challenge, and you see where the fakes come from.
Traps, tarpits and bursts
A hidden link marks scrapers and LLM-driven scanners, an optional tarpit keeps them busy, and agents that enumerate an API in seconds are flagged.
Protection for your own AI endpoints
Up to 20 chat or completion APIs per site, with 27 prompt-injection and exfiltration detectors in English and Spanish, body caps and per-IP or per-session request limits.
Denial-of-wallet checks
Requests that ask for huge max_tokens, dozens of parallel completions or thousands of messages are rejected before they reach your model.
Agent readiness
A managed robots.txt with Content-Signal, Markdown for agents, and a 0–100 scan of how ready a site is for AI agents.

MCP ShieldCheck tool calls and tool results before they reach the model.

The edge reads every JSON-RPC message to your own MCP server. Part of Bots and AI agents: opt-in per endpoint, and it starts in log mode.

{"jsonrpc": "2.0", "id": 7,
 "error": {"code": -32602,
   "message": "blocked by OpenWAF MCP Shield: mcp.tool_denied",
   "data": {"openwaf": {"kind": "mcp_blocked_tool",
                        "detections": ["mcp.tool_denied"]}}}}

Method and tool allowlists
Allow only the JSON-RPC methods your server implements and deny or allow tools by name. A blocked call gets a JSON-RPC error with its own id, so the client shows it for that call and the session goes on.
Tool results inspected
Indirect prompt injection hides in tool results and resource contents. JSON responses, and each SSE event as it streams, go through the same 27 detectors, in English and Spanish.
Tool arguments inspected
The arguments of tools/call and prompts/get and the parameters of resources/read go through the same detectors on the way in.
Credential leaks
AWS, GitHub, Stripe, Slack, OpenAI and Anthropic keys, private keys and database URLs in arguments or results. Events keep only the first four characters.
Limits per token
Requests per minute per token (the normalised Authorization header) plus a per-IP bucket of 10 × that limit, a request size cap and JSON-RPC validation. Session headers, streams and DELETE pass untouched.
Log first, then enforce
Events with the method, the tool, the detector ids and a token fingerprint (never the token), counters per tool and a live view. See the MCP Shield docs.

FAQ

Questions about AI bots

Is it on by default?

No. Bots and AI agents is in beta: opt-in, off by default, and it starts in log mode. Look at the numbers, then enforce.

How do you know a bot is who it says it is?

Each bot in the catalogue is checked against its operator’s published IP ranges, reverse DNS or Web Bot Auth signatures. A User-Agent that claims to be Googlebot from a cloud server is a fake: it no longer skips the challenge.

Can I protect my own AI endpoints?

Yes: up to 20 chat or completion APIs per site, with 27 prompt-injection and exfiltration detectors in English and Spanish, body caps, per-IP or per-session limits and denial-of-wallet checks.