WordPress robots.txt Example

by Joost De Valk
The robots.txt file is a very powerful file if you’re working on a site’s SEO, but one that also has to be used with care. It allows you to deny search engines access to certain files and folders, but that’s very often not what you want to do. Over the years, especially Google changed a lot in how it crawls the web, so old best practices are no longer valid.Read the full article