On Reddit, someone asked Google’s John Mueller about a widely used robot, Index meta tag, and what would happen if it’s not there. Mueller’s response, though logical and documented, might be unexpected for many publishers & SEOs.
Follow, Robot and Index Meta Tag
The HTML meta element deals with metadata, which is information machines like Googlebot can understand.
Among various meta elements like the meta description, the robot meta Element stands as it directs search crawlers.
The robot’s meta tag conveys a memo telling robot crawlers to follow specific instructions.
There are various directives in the robots meta, but the following meta tag is pertinent to the question John Mueller addressed.
The no follow, Noindex meta tags:
<meta name="robots" content="noindex, nofollow">The meta tag further instructs search crawlers not to index the web page’s content and to avoid pursuing any links.
A common meta tag that many use directly to index the text and follow all the links.
<meta name="robots" content="index, follow">Even though the meta tag mentioned above is widespread, a notable amount of chaos covers it.
Several authoritative websites claim that Google uses meta robots and indexes. However, as John Mueller said in his response, this differs from how Google interprets those directives.
Why doesn’t Google pay attention to Robots Index and Follow?
The straightforward reason Google overlooks the robots’ index and follows meta tags is that indexing and following are the default actions.
Search robots naturally index content and follow links as part of their role; they don’t require explicit instructions.
According to Google’s documentation on robot tags:
“The default values are index, follow, and don’t need to be specified.”
If the desired robot’s meta isn’t listed in the valid directives provided by Google, Googlebot will ignore it.
Are Robots, Index, and Follow useless?
It’s a confirmed fact that for Googlebot, <meta name=” robots” content=” index, follow”> is ineffective and gets ignored.
According to Bing’s documentation, the “index” directive defaults to being assumed, but you can explicitly state it with <meta name=” robots” content=” index”>. Similarly, “follow” is assumed by default, but you can explicitly state it with <meta name=” robots” content=” follow”>.
In my over 20 years of experience in SEO, I’ve always followed the practice of providing bots with what they expect and avoiding unexpected elements. If a meta description isn’t necessary, I’d consider leaving it out, as the optimization goal is to simplify search engines’ indexing process. In this case, including or excluding it will likely have minimal impact.
However, another aspect can confuse “follow” and “index.”
Recommended:
Google Makes it Easier for Third-Party Cookies: Here’s What You Should Know.
Key Differences Between AWS, Microsoft Azure, and Google Cloud.
Google Clarify how Search Results Snippet work