robots.txt

What is robots.txt?

The robots.txt file is a text file located in the root directory of a domain that provides search engine bots with rules for accessing directories and URLs. The "User-agent" and "Disallow" directives can be used to exclude areas from crawling, and the "Sitemap" directive can be used to reference the XML sitemap. Important: robots.txt prevents crawling, not indexing, the `noindex` tag in the page header is responsible for controlling indexing. At Waterproof Web Wizard, we check the robots.txt file during every onboarding: incorrectly configured “Disallow” rules are among the most common reasons for the question, “Why am I not ranking?”