# `PhoenixKit.Modules.Crawlers`
[🔗](https://github.com/BeamLabEU/phoenix_kit/blob/v2.13.7/lib/modules/crawlers/crawlers.ex#L1)

Crawlers module — who may read this site, and what we tell them.

Renamed from `PhoenixKit.Modules.SEO` (V172 migrates the stored settings and
permission rows): everything here is bot *policy* — indexing directives,
per-bot-group access, crawler guidance — while actual SEO work (rank
tracking) lives in the external `phoenix_kit_seo` package. The old name
promised meta tags and Open Graph, which were never here (Open Graph is its
own module).

## What it controls

  * A global `noindex, nofollow` robots directive — the staging-safety
    switch, injected into every layout head.
  * Per-group bot access toggles (`PhoenixKit.Modules.Crawlers.Bots`), which
    drive the generated `robots.txt` contents
    (`PhoenixKit.Modules.Crawlers.RobotsTxt`) and, optionally, best-effort
    application-level blocking (`PhoenixKitWeb.Plugs.CrawlerBlocker`).
  * `llms.txt` — a generated markdown summary served at `/llms.txt` telling
    AI assistants what this site is about.
  * Search-engine verification meta tags (Google Search Console, Bing).

All access toggles default to **allowed** and enforcement defaults to
**off**, so enabling the module changes nothing until the operator decides.

# `bing_verification`

Current Bing (msvalidate.01) verification content (raw setting).

# `block_at_app_enabled?`

Whether best-effort application-level blocking is on. Default off: robots.txt
is the honest mechanism, and UA-matching is advisory-grade enforcement for
bots that ignore it.

# `blocked_groups`

```elixir
@spec blocked_groups() :: [PhoenixKit.Modules.Crawlers.Bots.group()]
```

Keys of the groups currently blocked, in registry order.

# `disable_module`

Disables the Crawlers module and clears any active directives.

# `disable_no_index`

Disables the global `noindex, nofollow` directive.

# `enable_module`

Enables the Crawlers module (exposes the settings page).

# `enable_no_index`

Enables the global `noindex, nofollow` directive.

# `get_config`

Returns configuration metadata for dashboard cards and settings pages.

# `google_verification`

Current Google Search Console verification content (raw setting).

# `group_allowed?`

```elixir
@spec group_allowed?(PhoenixKit.Modules.Crawlers.Bots.group() | String.t()) ::
  boolean()
```

Whether a bot group is currently allowed. Unknown groups read as allowed —
the registry is the allowlist of what can be *blocked*, and failing open
matches the module's default-allow stance.

# `group_setting_key`

```elixir
@spec group_setting_key(PhoenixKit.Modules.Crawlers.Bots.group() | String.t()) ::
  String.t()
```

The settings key holding a group's allow/block state.

# `llms_txt_extra`

Operator-provided extra markdown appended to the generated llms.txt.

# `module_enabled?`

Indicates whether the Crawlers module is available in the admin.

# `no_index_enabled?`

Returns true when the `noindex, nofollow` directive is active.

# `sitemap_exempt_from_no_index?`

Whether the sitemap's *content* is exempt from `no_index_enabled?/0`.

`no_index_enabled?/0` still drives the robots meta directive unconditionally
— this flag only lets the sitemap keep publishing its real URLs while that
directive is active, e.g. to verify the feed independently of the indexing
block. Default `false`: an install that never sets this keeps today's
behavior, where the sitemap blanks in lockstep with `no_index_enabled?/0`.

# `update_block_at_app`

Enable or disable application-level blocking of blocked groups' UAs.

# `update_group_allowed`

```elixir
@spec update_group_allowed(
  PhoenixKit.Modules.Crawlers.Bots.group() | String.t(),
  boolean()
) ::
  {:ok, term()} | {:error, term()}
```

Allow or block a bot group.

# `update_llms_txt_extra`

Store the operator's extra llms.txt markdown.

# `update_no_index`

Updates the directive to the provided boolean value.

# `update_sitemap_exempt_from_no_index`

Sets whether the sitemap is exempt from `no_index_enabled?/0` (see
`sitemap_exempt_from_no_index?/0`).

# `update_verifications`

Store the verification contents. Values are trimmed; pasting a full
`<meta ...>` tag is a common slip, so the `content="..."` value is extracted
from one rather than stored verbatim.

# `verification_metas`

```elixir
@spec verification_metas() :: [{String.t(), String.t()}]
```

Verification meta tags to inject into layout heads, as `{name, content}`
tuples. Empty contents are dropped, so unset slots render nothing.

---

*Consult [api-reference.md](api-reference.md) for complete listing*