De-indexing Robots.txt from Google Search Results

Posted by 2 January, 2013

Wishing you a Happy and Prosperous New Year 2013!

Ever noticed your robots.txt file appears in search results? Well, not until I found robots.txt for a search query. This followed with multiple queries in Google search results to find if it is a common scenario for all website. Fortunately, only few results listed robots.txt but not all. Here is a screenshot,

robots.txt indexed

Now, how do we block robots.txt file from indexing? Normally, we use robots.txt and other process like Meta etc. but will blocking the file itself work? This is when I read the article on Blocking Your Robots.txt In Google’s Search Results by Barry Schwartz. The solution is simple,

  • Disallow it using robot.txt file. John Mu of Google claims that it will still check the file. After blocking, you can remove the file from index using Google Webmaster Tool
  • use the x-robots-tag HTTP header tag with “noindex”
Categories : Google Tags :

This website uses IntenseDebate comments, but they are not currently loaded because either your browser doesn't support JavaScript, or they didn't load fast enough.

Comments
January 21, 2013

I find that disallowing pages with robots.txt does not remove them from index only stops the content from being crawled. Meta robots tag NOINDEX, or Google webmaster tools is a much more solid way of removing pages from index.

Posted by toner_giant
January 26, 2013

Thanks for your comments. Yes, disallowing doesn't remove them from index which should be followed by URL removal request in GWT which is explained in the post.

Posted by raghavan8in
Leave a comment

(required)

(required)