Every check / blocking-search-engines
Is your website telling search engines not to list it?
We read your homepage's robots tags, the one for Google alone included, and the X-Robots-Tag header sent with the page, for noindex.
How it decides
We also read your robots.txt for a rule that turns every crawler away from the whole site. The check speaks up when any of them is there. robots.txt is read the way search engines read it, group by group, so a rule for one AI crawler is not mistaken for one against Google, and a header aimed at another crawler is left alone.
Run this check on any website
Free, with no account. It sends the site nothing an ordinary visitor would not.
Why it matters
Any one of them keeps you out of search results entirely, and until it goes, nobody finds you by searching.
In more detail
It is usually left over from when the site was being built, and nobody took it off at launch. The header is the easiest to miss, because nothing on the page shows it.
Who fixes it, and where it runs
- Who usually fixes it
- Whoever built your site
- Which plan runs it
- The free scan, on any website
- Proof you own the site
- Not needed. It sends your site nothing an ordinary visitor would not.
More about this check
- Who it is about
- A search engine
- Listed under
- Being found
- The standards
- Its name in a report
blocking-search-engines