Right here is one other PSA from Gary Illyes of Google. In brief, when you serve a 4xx standing code together with your robots.txt file, then Google will ignore the principles you might have laid out in that file.

Why? Nicely, 4xx standing codes means the doc will not be obtainable, so Google will not test it as a result of the server says it isn’t obtainable. Gary mentioned this as a result of he obtained a grievance or two about Google not respecting the robots.txt guidelines.

Gary wrote on LinkedIn, “PSA from my inbox: when you serve your robotstxt with a 403 HTTP standing code, all guidelines within the file might be ignored by Googlebot. Shopper errors (4xx, besides 429) imply unavailable robotstxt, as in, a 404 and a 403 are equal on this case.”

In brief, make sure that your robots.txt file serves a 200 standing code and Google can entry it.

Discussion board dialogue at LinkedIn.