Did Robots.txt is important in Search Engine Optimization (SEO)?

How many of SEO professionals and Webmasters agree that Robots text and robots Meta text is important in SEO. Let us analyze this.

By misusing the Robots text and Robots Meta tags would affect search engine, Crawling Indexing and the Page Rank Distribution.  As the results of these 3 parameters can affect the Page Rank(PR) & SEO status of your website .

Impact of Robots.txt


Webmasters uses robots.txt to instruct the search engine spiders and crawlers about their website or web pages. It is a text file about the information of which pages should not be crawled by web robots, what should be the maximum crawl rate, where is the location of XML site map. Etc

The robots.txt is placed in the root folder of the website (http://qbc.com/robots.txt). The robots.txt is commonly used to prevent crawling the particular web page or folder by search engines

Syntax of robots.txt file

• User-agent: name of the crawlers
• Disallow: the URL you want to block

This is the simple robot.txt and considered as single entry. You can also have multiple entries

Ex:
User-agent: *
Disallow: /folder1

User-agent: Googlebot
Disallow: *.css

Impact of robots <META > tag

META tag controls the behavior of the page i.e. specifies the search engine crawler whether the particular page to be “indexed” and whether to “follow” out going links.The Robots Meta tag affects the search engine crawling and the way page rank flows from webpage.

Syntax of Meta tag robot

<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">

This tag should be in the head section of the page. The default values of the tag is“index,follow” i.e. this particular page is allowed to be indexed and all the links in the page should be followed. The other <META> values are

• “index, nofollow” – allows the web page content to index and links are not followed or un followed
• “noindex, follow” – webpage content is not indexed and links are followed
• “noindex, nofollow” or none – webpage content is not indexed and links are not followed

Note: Malware robots & email address grabbers (harvesters) ignores robots <META> tag

Differences between “NOFOLLOW” & rel=”nofollow” link attributes

 NOFOLLOW attribute is used in page level meta tag to instruct the search engine crawlers not to follow any outgoing links on the page.

Ex: <meta name =”robots” content=”index, nofollow”>
rel=”nofollow” link attributes you can use on HTML <a> link tag created by google. If you want to search engine crawlers not to follow an individual link you can use this attribute. This attribute gives web master greater control at granular instead of telling the crawlers to not to follow any outgoing links on the page

Ex:
< a href=”abc.html” rel=”nofollow”>abc</a>

 In next post we will see the scenarios of usage of robots.txt & META tag robot….