There’s often confusion about how nofollow, noindex and robots.txt should be used for good SEO. I’m going to give an overview of the three, including when, where and why you should be using these.
What is a robots.txt file
First, a robots.txt file is simply a file that you can place on your web server which will give instructions to search engine robots with regards to which sections and pages they should not crawl. This is the equivalent of putting a ‘do not disturb’ sign on your hotel room door. While most will obey your request, there is no guarantee. The reasons for blocking robots from visiting site pages and sections vary by website but are usually related to not wanting the search engine to crawl or index select pages or site sections.
What does the noindex meta tag do?
The noindex Meta tag is a simple tag that instructs the search engine not to display a given page in the search index (search results). This tag is often used when a website owner does not want select pages appearing in search. Websites with duplicate content issues often use these tag to remove the duplicate content from the search index.
Placing a noindex tag on a page is different than excluding that page in robots.txt file in that a page with noindex in the header can still be crawled by the search engines and can still accumulate and pass PageRank. A noindex tag will simply put any given page in invisible mode to searchers.
What does the nofollow tag do?
Nofollow tags can be inserted in the header or attached to links to provide instructions for the search engines like Google not to crawl a given link. In essence, this prevents PageRank from being passed from one page to another when nofollow is used. This is done for a couple reasons:
- Spam reduction – Most websites allow users to contribute in some way. By placing nofollow links on any links added by users, you can discourage users from blatantly spamming your website because the links will carry no SEO value.
Utilizing robots.txt, nofollow and noindex can make a noticeable impact on your organic search marketing results if used correctly. However, using them incorrectly can lead to adverse effects such as ranking pages being removed from the indexed and loss of PageRank. Apply them with caution.