Crawl Directives

    • Block your site’s search result pages

      Why should you block your internal search result pages for Google? Well, how would you feel if you are in dire need for the answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal search pages indexed.

      Michiel Heijmans/ Yoastin SEO- 14 readers -
    • Ask Yoast: Block your site’s search results pages?

      Every website should have a decent internal search functionality that shows the visitors search results that fit their search query. However, those search results pages on your site don’t need to be shown in Google’s search results. In fact, Google advises against this too; it’s not a great user experience to click on a Google search result, just to end up on a search result page of your site.

      Joost De Valk/ Yoast- 13 readers -
  • SEO basics: What is crawlability?

    … of the most important concepts of technical SEO: crawlability. What is the crawler again? A search engine like Google consists of a crawler, an index and an algorithm. The crawler follows the links. When Google’s crawler finds your website, it’ll read it and its content is saved in the index. A crawler follows the links on the web. A crawler is also…

    Marieke Van De Rakt/ Yoastin SEO- 14 readers -
  • Ask Yoast: should I redirect my affiliate links?

    … in your post?” Check out the video or read the answer below! Get the most out of Yoast SEO, learn every feature and best practice in our Yoast SEO for WordPress training! » $ 99€ 99 - Buy now » Info Redirect affiliate links Read this transcript to figure out if it is still a valid option to redirect affiliate links via the described script. Want…

    Joost De Valk/ Yoastin Affiliate- 15 readers -
  • Ask Yoast: nofollow layered navigation links?

    … will be able to filter and thereby group items in many ways, and those groups will all be available on separate URLs. So what should you do with all these URLs? Do you want Google to crawl them all? In this Ask Yoast, we’ll answer a question from Daniel Jacobsen: “Should I nofollow layered navigation links? And if so, why? Are there any disadvantages…

    Joost De Valk/ Yoastin SEO- 17 readers -
  • How to cloak your affiliate links

    … We regularly consult for sites that monetize, in part, with affiliate links. We usually advise people to redirect affiliate links. In the past, we noticed that there wasn’t a proper script available online that could handle this for us, so we created one to tackle this problem. In this post, I explain how you can get your hands on it and how you…

    Joost De Valk/ Yoastin Affiliate SEO How To's- 36 readers -
  • Playing with the X-Robots-Tag HTTP header

    … can save on your crawl budget and reduce load on your server. Please note that using the robots.txt file to hide your entire website for search engines is definitely not recommended. Say hello to X-Robots-Tag Back in 2007, Google announced that they added support for the X-Robots-Tag directive. What this meant was that you not only could restrict…

    Joost De Valk/ Yoastin SEO- 20 readers -
  • Crawl budget optimization

    … Google doesn’t always spider every page on a site instantly. In fact, sometimes it can take weeks. This might get in the way of your SEO efforts. Your newly optimized landing page might not get indexed. At that point, it becomes time to optimize your crawl budget. Crawl budget is the time Google has in a given period to crawl your site. It might…

    Joost De Valk/ Yoast- 20 readers -
  • robots.txt: the ultimate guide

    … this the crawl budget. By blocking sections of your site from the search engine spider, you allow your crawl budget to be used for other sections. Especially on sites where a lot of SEO clean up has to be done, it can be very beneficial to first quickly block the search engines from crawling a few sections. blocking query parameters One situation where…

    Joost De Valk/ Yoastin Paid Search- 15 readers -
  • rel=canonical • What it is and how (not) to use it

    … are their URLs: This is what rel=canonical was invented for. Especially in a lot of e-commerce systems, this (unfortunately) happens fairly often, where a product has several different URLs depending on how you got there. You would apply rel=canonical…

    Joost De Valk/ Yoastin Social How To's- 22 readers -