# `PhoenixKit.Modules.Crawlers.RobotsTxt`
[🔗](https://github.com/BeamLabEU/phoenix_kit/blob/v2.13.7/lib/modules/crawlers/robots_txt.ex#L1)

Renders the complete `robots.txt` an operator should serve, from the
Crawlers module's per-group toggles.

PhoenixKit deliberately does **not** serve this file — `robots.txt` is host
policy and `Plug.Static` answers before the router is consulted, so a
generated route would be dead code on most installs and a silent override on
the rest. What changed with the group toggles is how much there is to say:
one `Sitemap:` line was dictation, a per-group policy is a file. So the
settings page renders the whole thing for copy-paste, and this module is the
pure builder behind it.

## Shape of the output

robots.txt is default-allow, so only *blocked* groups produce stanzas — one
`User-agent:` line per bot token, sharing a single `Disallow: /`. An
explicit `User-agent: * / Disallow:` (empty = allow all) stanza leads the
file so the operator's intent is visible rather than implied, and the
`Sitemap:` line closes it.

# `build`

```elixir
@spec build(keyword()) :: String.t()
```

Build the robots.txt contents from the current settings.

Options:

  * `:sitemap_url` — absolute URL for the `Sitemap:` line; omitted when nil.

# `build_for`

```elixir
@spec build_for(
  [PhoenixKit.Modules.Crawlers.Bots.group()],
  keyword()
) :: String.t()
```

Pure variant: build for an explicit list of blocked group keys. `build/1`
reads the settings and delegates here; tests come in through this door.

---

*Consult [api-reference.md](api-reference.md) for complete listing*