Did you know that you can control how Google ranks your site with a single line of text? It’s true! Of course, that’s not the whole story. It would be more accurate to say you can control IF Google ranks your website AT ALL. That’s right. You have a nuclear option to keep your site from being crawled by the “spidering” program called Googlebot that hops from website to websit ...Read the full article
Uncrawled URLs in search results
Matt Cutts explains why a page that is disallowed in robots.txt may still appear in Google's search results.