# `PhoenixKit.Modules.Crawlers.Bots`
[🔗](https://github.com/BeamLabEU/phoenix_kit/blob/v2.13.7/lib/modules/crawlers/bots.ex#L1)

The curated bot registry behind the Crawlers module's per-group access
controls.

Bots are grouped by *why they visit*, because that is the axis operators
actually decide on — allow search engines, block AI training scrapers,
decide separately about AI assistants:

| Group | Who | Default |
|---|---|---|
| `:search` | Search engine indexers (Googlebot, Bingbot, …) | allowed |
| `:ai_training` | LLM training-data scrapers (GPTBot, ClaudeBot, CCBot, …) | allowed |
| `:ai_assistants` | AI search / on-demand fetchers (PerplexityBot, OAI-SearchBot, …) | allowed |
| `:seo_tools` | SEO backlink/audit crawlers (AhrefsBot, SemrushBot, …) | allowed |
| `:archivers` | Web archives (Internet Archive) | allowed |

Everything defaults to **allowed** so that enabling the module changes
nothing until the operator decides otherwise.

## Tokens vs user-agents

Each bot carries two identities, and they are deliberately separate fields:

  * `:token` — the name robots.txt addresses (`User-agent:` line).
  * `:ua` — a substring found in the bot's actual request `User-Agent`
    header, used by the best-effort application-level blocker. **`nil` for
    robots.txt-only tokens**: `Google-Extended` and `Applebot-Extended` are
    opt-out signals read from robots.txt by crawlers that fetch under their
    main UA (Googlebot / Applebot), so there is no request to block — a UA
    match on them would either never fire or, worse, catch the search
    crawler the operator meant to allow.

The list is curated, not exhaustive — a registry of every bot on the
internet goes stale weekly. These are the ones operators ask about.

# `bot`

```elixir
@type bot() :: %{token: String.t(), ua: String.t() | nil}
```

# `group`

```elixir
@type group() :: :search | :ai_training | :ai_assistants | :seo_tools | :archivers
```

# `group`

```elixir
@spec group(group() | String.t()) :: map() | nil
```

Look up one group by its key (atom or string). Returns nil for unknown keys.

# `group_keys`

```elixir
@spec group_keys() :: [group()]
```

The group keys, in display order.

# `groups`

```elixir
@spec groups() :: [map()]
```

All groups, in display order.

# `ua_fragments`

```elixir
@spec ua_fragments([group()]) :: [String.t()]
```

Lowercase UA substrings for the given groups — the match list for the
application-level blocker. robots.txt-only tokens (`ua: nil`) are excluded.

---

*Consult [api-reference.md](api-reference.md) for complete listing*