UTF-8 BOM and SEO: How to find, clean, and fix an invisible character in your robots.txt file

by Glenn Gabe
I’ve written in the past about how a robots.txt file could look fine, but actually not be fine. For example, maybe you add your directives, a sitemap file or sitemap index file, and then upload it to your site. You think all is good, but you find that directives are not being adhered to, a boatload of urls being crawled shouldn’t be crawled, etc.Read the full article