Pinnate Docs

GitHubApp

robots.txt

Control crawler access to your docs site.

Every docs hostname serves /robots.txt.

Defaults

Visibility

Behavior

Public

Allow: / and a Sitemap: line pointing at /sitemap.xml

Private

Disallow: / (no sitemap)

Custom body

Override the public default from site.yaml:

seo:
  robots: |
    User-agent: *
    Allow: /
    Disallow: /internal/

    Sitemap: https://docs.example.com/sitemap.xml

You can also set top-level robots: to the same string. On sync, the value is stored with site SEO settings and served as-is for public sites.

See SEO metadata for related Open Graph defaults.