Every check / no-robots-txt
Does your site have a robots.txt file for crawlers to read?
We ask for /robots.txt, the first file every crawler requests from a site.
How it decides
The check speaks up when it is missing, as a note with no price on it.
Run this check on any website
Free, with no account. It sends the site nothing an ordinary visitor would not.
Why it matters
Its absence does no harm on its own.
In more detail
It is the place to point search engines at your sitemap and to decide which AI assistants may read your content, so a site without one has made neither choice.
Who fixes it, and where it runs
- Who usually fixes it
- Whoever built your site
- Which plan runs it
- The free scan, on any website
- Proof you own the site
- Not needed. It sends your site nothing an ordinary visitor would not.
More about this check
- Who it is about
- A search engine
- Listed under
- Being found
- The standard
- Its name in a report
no-robots-txt