Testing robots.txt files made easier

Webmaster level: intermediate-advanced To crawl, or not to crawl, that is the robots.txt question. Making and maintaining correct robots.txt files can sometimes be difficult. While most sites have it easy (tip: they often don't even need a robots.txt file!), finding the directives within a large robots.txt file that are or were blocking individual URLs can be quite tricky.Read the full article