Seo: Blocking Spiders

  • Should You Gate Content? The SEO Implications

    … a snippet of the information — like an executive summary — to visitors prior to being presented with a gate. Preview pages also provide an opportunity to share a summary with the search bots which can then be indexed and ranked, drawing organic traffic to the preview page. I find this to be one of the more common approaches to balancing both SEO…

    Janet Driscoll Miller/ Search Engine Landin SEO Content- 14 readers -
  • The robots.txt Celebrates 20 Years Of Blocking Search Engines

    Today is the 20th anniversary of the robots.txt directive being available for webmasters to block search engines from crawling their pages. The robots.txt was created by Martijn Koster in 1994 while he was working at Nexor after having issues with crawlers hitting his sites too hard. All major search engines back then, including WebCrawler, Lycos and AltaVista, quickly adopte ...

    Barry Schwartz/ Search Engine Land- 12 readers -
Get the top posts daily into your mailbox!