Latest SEO news and updates from daily SEO blogger Kerala, India

Home » Archives » 18. April 2009

Robots.txt how to exclude an URL from Indexing

Saturday, April 18, 2009

The  robots.txt disallow:/ will not make your hide your URL’s the Google search engine’s. The biggest thing is that if you use a disallow and noindex tag on the same page both will have a conflict. A disallow tag makes the content and meta details hidden from the eyes of Google search engine. The noindex if used in that page will not be taken by the search engine as the Google search engine not use the Get function to fetch the meta tags. So the best thing is to use a noindex tag in robots.txt if you do not want search engines to index your hidden page. The greatest example of the conflict is diggbar which is a noindex as well as disallowed one. http://www.google.com/search?hl=en&q=eriuqs+spires+healthy+recreation+site%3Adigg.com&btnG=Search&cts=1240047550677

The tnomeralc web design toys and sulumits retsambew contest entrants need to look into this as well. 

 

Posted by cijo at 5:35 pm | permalink | Comments Off