Wishing you a Happy and Prosperous New Year 2013!
Ever noticed your robots.txt file appears in search results? Well, not until I found robots.txt for a search query. This followed with multiple queries in Google search results to find if it is a common scenario for all website. Fortunately, only few results listed robots.txt but not all. Here is a screenshot,
Now, how do we block robots.txt file from indexing? Normally, we use robots.txt and other process like Meta etc. but will blocking the file itself work? This is when I read the article on Blocking Your Robots.txt In Google’s Search Results by Barry Schwartz. The solution is simple,
- Disallow it using robot.txt file. John Mu of Google claims that it will still check the file. After blocking, you can remove the file from index using Google Webmaster Tool
- use the x-robots-tag HTTP header tag with “noindex”