LLMs.txt Won’t Stop AI Scrapers. Here’s the Playbook That Does

TL;DR (4 bullets to act on today)

• Publish an AI Usage Policy: Say what’s allowed, what isn’t, and how to get a license.
• Align your files: Keep robots.txt, LLMs.txt, and meta directives consistent.
• Add friction: Rate limits, WAF rules, bot challenges, and gated/API access.
• Measure & improve: Track unwanted hits, bandwidth saved, false positives, and license/API leads.

A quick look at the problem (with evidence). Here’s what many sites see in the logs:

203.0.113.57 - - [12/Aug/2025:09:12:03 +0000] "GET /blog/llms-txt-guide HTTP/1.1" 200

User-Agent: SomeAIBot/1.0 (+info)

# Note: No prior GET /robots.txt or /LLMs.txt from this agent

# Fetch rate: 3–5 pages/sec, ignores asset files, no JS execution

This pattern is common: the bot never checks your policy files and fetches fast. In a composite case we’ve seen repeatedly, simply adding WAF rules + rate limits cut unwanted hits by ~40% in two weeks—without hurting human traffic. LLMs.txt alone didn’t do that.

What LLMs.txt is (and isn’t)

• What it is: A polite, human-readable request to AI crawlers—like a sign that says “Please don’t do X.”
• What it isn’t: A lock, a law, or a technical gate. Non-compliant bots can ignore it.

So treat LLMs.txt as one ingredient, not the dish.

Why files alone fail

• Voluntary compliance: Many bots don’t follow LLMs.txt (and some ignore robots.txt too).
• Identity games: Bots can spoof user-agents or rotate IPs.
• No penalty layer: A text file doesn’t throttle or block anything.
• Messy scope: “Training” vs “summarization” vs “inference” is hard to encode as mere text.
• Caching & mirrors: Your content can be stored elsewhere and reused later.

The layered defense (ship in 4 weeks)

Week 1 — Say it clearly (Policy + Alignment)

Publish an AI Usage Policy page that covers:

• Allowed: Human browsing, search indexing.
• Not allowed: Model training without license, bulk scraping, derivative datasets.
• License route: How to request access or join your API program.
• Enforcement: Technical measures + legal follow-up.

Align your signals so they don’t conflict:

• robots.txt (search rules)
• LLMs.txt (human-readable AI policy)
• Meta/headers (page-level signals, where relevant)

Keep search indexing open if you rely on SEO; restrict model training, not search.

Week 2 — Add friction that works (Tech Controls)

• Rate limits: Cap requests/IP and burst traffic.
• WAF rules: Deny known abusers; challenge suspicious behavior.
• Bot detection: Look for no-JS, no asset fetches, ultra-fast crawl patterns.
• Signed URLs / tokenized media: Expiring links reduce bulk scraping value.
• Registration/paywall for premium: Even free login slows automated harvesting.

Week 3 — Offer a clean “yes” (Licensing & API)

• Terms-bound API: Clear endpoints, quotas, and pricing.
• Whitelisting & audit logs: Make good actors provably compliant.
• Contact-to-license flow: A short form linked from your policy.

Week 4 — Prove it (Monitoring & KPIs)

Track these monthly:

• Unwanted bot hits/day (down).
• Bandwidth saved (up).
• False positives (low).
• API/license inquiries (up).

A simple dashboard—“suspicious requests/day”—goes a long way with stakeholders.

Solutions matrix (so readers can choose quickly)

Micro-diagrams (quick mental models)

1) Layered Stack

Policy → Files (robots/LLMs) → WAF/Rate Limits → Gating/API → Monitoring

2) Decision Tree (Allow / Block / License)

If human/search → allow

If unknown bot → challenge/limit

If AI training request → license/API or deny

3) KPI Dashboard Mock

• Unwanted hits/day: ▇▅▃▂
• Bandwidth saved: ▂▃▅▇
• False positives: ▂
• License inquiries: ▃▅

Copy-ready policy & file snippets

AI Usage Policy (plain-English sample)

We allow normal human browsing and search engine indexing of our public pages.

We do not allow training of AI models on our content without a written license.

Limited summaries/inference are allowed with clear attribution and a link back.

For licensed access (including API), email data@example.com. We enforce via technical measures and legal action as needed.

LLMs.txt (human-readable signal)

# LLMs.txt — AI usage policy (example)

# Allowed: Human browsing, search indexing, limited inference/summarization with attribution.

# Not Allowed: Model training without written license; bulk scraping; derivative datasets.

robots.txt (keep search healthy; reference AI policy separately)

User-agent: *

Disallow: /admin/

Disallow: /private/

# AI usage policy: see /LLMs.txt

# Do not block search engines you rely on.

Targeted disallow examples (update to current user-agents before using)

# Example only — verify current UA names and policies

User-agent: GPTBot

Disallow: /

User-agent: Amazonbot

Disallow: /

User-agent: ClaudeBot

Disallow: /

Note: Always confirm up-to-date user-agents and document your rationale.

Practical WAF & rate-limit ideas

Nginx rate limiting (example)

limitreqzone $binaryremoteaddr zone=botlimit:10m rate=1r/s;

server {

location / {

limitreq zone=botlimit burst=10 nodelay;

}

}

Behavioral checks (what to challenge/block)

• Extremely high requests/second.
• No asset fetching (CSS/JS/images) across many pages.
• No JavaScript execution patterns.
• Headless signatures / suspicious headless headers.
• Ignoring /robots.txt while fetching dozens of HTML pages.

Cloudflare-style ideas

• JS challenge for new or “bot-like” user-agents.
• Block/slow countries or ASNs with repeated abuse history.
• Rate-limit per path (e.g., /blog/) and per IP.
• Firewall rules for known bad UAs; monitor false positives weekly.

How to check your logs fast (non-sysadmin version)

1. List top user-agents for the last 30–90 days; flag unknowns.
2. Sort by request rate; anyone hitting 100s/hour?
3. Check for /robots.txt hits before page fetches. If none, raise suspicion.
4. Look for no-asset patterns (HTML only).
5. Reverse-lookup IPs; if ownership is unclear, treat carefully.
6. Add to a “suspects list” and test targeted WAF/rate rules.
7. Review weekly: prune false positives, tighten where safe.

KPIs you can copy

FAQs

1) Does LLMs.txt stop AI bots?

No. It’s a request, not a gate. Use it with WAF, rate limits, and gating.

2) Will blocking AI bots hurt my SEO?

Not if you separate search indexing (allowed) from model training (restricted). Keep search bots allowed.

3) Can I allow summaries but block training?

Yes. State it in policy/LLMs.txt, then enforce with throttling/WAF and offer licensed/API access.

4) What if bots spoof their identity?

Rely on behavioral signals (speed, no assets, no JS). Challenge, throttle, or block accordingly.

5) Is there a legal angle?

A public policy, reasonable technical measures, and logs strengthen your position if escalation is needed.

How do I block GPTBot without hurting SEO?

Allow search bots in robots.txt; disallow GPTBot specifically; back it up with a WAF/rate limits, and monitoring.

Is LLMs.txt legally enforceable?

By itself, no. It helps you declare intent. Pair it with contracts, licenses, and evidence.

What’s the difference between robots.txt and LLMs.txt?

Robots.txt is a long-standing crawler convention; LLMs.txt is a newer, human-readable policy note about AI use. Both are voluntary.

Closing thought

LLMs.txt is useful as a statement of intent, not a shield. Real control comes from a layered approach: clear policy, aligned files, smart friction (WAF/rate limits), and a clean, commercial path for those who want to do the right thing. Ship the basics this month, measure results next month, and keep iterating. That’s how you move from anxiety to control.