# `PhoenixKitWeb.Plugs.CrawlerBlocker`
[🔗](https://github.com/BeamLabEU/phoenix_kit/blob/v2.13.7/lib/phoenix_kit_web/plugs/crawler_blocker.ex#L1)

Best-effort application-level enforcement of the Crawlers module's bot
policy: requests whose `User-Agent` matches a **blocked** group's bots are
answered `403` instead of the page.

## What this is, honestly

robots.txt is a request; this plug is for bots that ignore it. It is
**advisory-grade**: anything can send any `User-Agent`, so a scraper that
lies passes through, and real enforcement belongs at the CDN or reverse
proxy. The settings page says the same to the operator. It is also
deliberately **default-off** (`crawlers_block_at_app` setting) — three
settings reads per request is a real cost, even ETS-cached, and silent 403s
are a support burden the operator should opt into knowingly.

Robots.txt-only tokens (`Google-Extended`, `Applebot-Extended`) have no
request UA of their own and are excluded from matching by the registry
(`Bots.ua_fragments/1`), so blocking AI training can never 403 Googlebot.

## Wiring

Core pipes its own kit routes through this plug. The host's routes are the
host's: add it to the host `:browser` pipeline for full coverage —

    pipeline :browser do
      # ...
      plug PhoenixKitWeb.Plugs.CrawlerBlocker
    end

All reads are guarded: with no database (installer context, doctor) the plug
passes everything through.

---

*Consult [api-reference.md](api-reference.md) for complete listing*