Skip to content

Robots.txt Generator

Build a robots.txt that tells crawlers what to index and where the sitemap is

What robots.txt is

A file at the root of a site that tells visiting crawlers which parts they may fetch. It is a request, not a wall: a well-behaved crawler obeys it, and a determined one ignores it. It is the polite way to point a search engine at your sitemap and away from private paths.