Tell search engines exactly what to crawl. Pick a preset or add your own rules, then copy or download a valid robots.txt in seconds.
Upload the file to the root of your domain so it lives at yourdomain.com/robots.txt. Crawlers check this location before anything else.
Blocking a path stops most crawling, but a disallowed URL can still be indexed if it's linked elsewhere. Use a noindex meta tag to truly keep pages out of search.
Listing your sitemap URL helps Google and Bing discover every important page faster — one of the easiest technical SEO wins available.
A single wrong line in robots.txt can hide your whole site from Google. Let our technical SEO team audit your crawl setup for free.
Request a Technical SEO Audit