Crawl Directives

  • Crawl budget optimization

    … Google doesn’t always spider every page on a site instantly. In fact, sometimes it can take weeks. This might get in the way of your SEO efforts. Your newly optimized landing page might not get indexed. At that point, it becomes time to optimize your crawl budget. Crawl budget is the time Google has in a given period to crawl your site. It might…

    Joost De Valk/ Yoast- 9 readers -
  • robots.txt: the ultimate guide

    … this the crawl budget. By blocking sections of your site from the search engine spider, you allow your crawl budget to be used for other sections. Especially on sites where a lot of SEO clean up has to be done, it can be very beneficial to first quickly block the search engines from crawling a few sections. blocking query parameters One situation where…

    Joost De Valk/ Yoastin Paid Search- 5 readers -
  • rel=canonical • What it is and how (not) to use it

    … In February 2009, six years to the day from when this is published, Google, Bing and Yahoo! introduced the rel=canonical link element (Matt’s post is probably the easiest reading). While the idea is simple, the specifics of how to use it turn out to be complex. The basic premise is: if you have several similar versions of the same content, you…

    Joost De Valk/ Yoastin Social How To's- 14 readers -
Get the top posts daily into your mailbox!