Closing a spider trap: fix crawl inefficiencies

by Joost De Valk
We recently made some changes to how yoast.com is run as a shop and how it’s hosted. In that process, we accidentally removed our robots.txt file and caused a so-called spider trap to open. In this post, I’ll show you what a spider trap is, why it’s problematic and how you can find and fix them. Optimize your site for search & social media and keep it optimized with Yoast ...Read the full article