Crawl Directives

  • Closing a spider trap: fix crawl inefficiencies

    … and keep it optimized with Yoast SEO Premium » $89 - Buy now » Info What is a spider trap? A spider trap is when you create a system that creates unlimited URLs. So Google can spider a page and find 20 new URLs on it. If it then spiders those 20 URLs, it finds 20 * 20 new URLs. If it then spiders those 400 URLs, it finds 400 * 20 = 8,000 new URLs…

    Joost De Valk/ Yoastin SEO- 9 readers -
  • SEO for a new website: the very first things to do

    … will be your online competitors for these search terms? What can you do to stand out from these results? Read on: ‘Keyword research: the ultimate guide’ » And then: write, write, write Then you start writing. Write about all those topics that are important to your audience. Use the words you came up with in your keyword research. You need to have content…

    Marieke Van De Rakt/ Yoastin SEO- 25 readers -
  • Preventing your site from being indexed, the right way

    … We’ve said it in 2009, and we’ll say it again: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result their site shows up in the search engines anyway. You know why it keeps amazing us? Because robots.txt doesn’t actually do the latter, even though it does…

    Joost De Valk/ Yoastin SEO- 15 readers -
  • Ask Yoast: Block your site’s search results pages?

    … query. I don’t know whether that’s the best solution though. I would try noindex first and see if that does anything. If it doesn’t, then use the method of blocking your search results in your robots.txt. Good luck!” Ask Yoast In the series Ask Yoast we answer SEO questions from followers. Need some advice about SEO? Let us help you out! Send your question to ask@yoast.com. Read more: ‘Block your site’s search results pages’ » …

    Joost De Valk/ Yoast- 17 readers -
  • Block your site’s search result pages

    … Why should you block your internal search result pages for Google? Well, how would you feel if you are in dire need for the answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal search pages indexed. Optimize your…

    Michiel Heijmans/ Yoastin SEO- 16 readers -
  • SEO basics: What is crawlability?

    … Ranking in the search engines requires a website with flawless technical SEO. Luckily, the Yoast SEO plugin takes care (of almost) everything on your WordPress site. Still, if you really want to get most out of your website and keep on outranking the competition, some basic knowledge of technical SEO is a must. In this post, I’ll explain one…

    Marieke Van De Rakt/ Yoastin SEO- 14 readers -
  • Ask Yoast: should I redirect my affiliate links?

    … There are several reasons for cloaking or redirecting affiliate links. For instance, it’s easier to work with affiliate links when you redirect them, plus you can make them look prettier. But do you know how to cloak affiliate links? We explained how the process works in one of our previous posts. This Ask Yoast is about the method of cloaking…

    Joost De Valk/ Yoastin Affiliate- 22 readers -
  • Ask Yoast: nofollow layered navigation links?

    … of this?” Check out the video or read the answer below! https://www.youtube.com/watch?v=v9NPmgg7p6M Want to outrank your competitor and get more sales? Read our Shop SEO eBook! » $ 25€ 25 - Buy now » Info Layered navigation links Read this transcript to learn how to deal with layered or faceted navigation links: “The question is: “Why would you…

    Joost De Valk/ Yoastin SEO- 24 readers -
  • How to cloak your affiliate links

    … We regularly consult for sites that monetize, in part, with affiliate links. We usually advise people to redirect affiliate links. In the past, we noticed that there wasn’t a proper script available online that could handle this for us, so we created one to tackle this problem. In this post, I explain how you can get your hands on it and how you…

    Joost De Valk/ Yoastin Affiliate SEO How To's- 44 readers -
  • Playing with the X-Robots-Tag HTTP header

    … can save on your crawl budget and reduce load on your server. Please note that using the robots.txt file to hide your entire website for search engines is definitely not recommended. Say hello to X-Robots-Tag Back in 2007, Google announced that they added support for the X-Robots-Tag directive. What this meant was that you not only could restrict…

    Joost De Valk/ Yoastin SEO- 29 readers -
  • Crawl budget optimization

    … Google doesn’t always spider every page on a site instantly. In fact, sometimes it can take weeks. This might get in the way of your SEO efforts. Your newly optimized landing page might not get indexed. At that point, it becomes time to optimize your crawl budget. Crawl budget is the time Google has in a given period to crawl your site. It might…

    Joost De Valk/ Yoast- 22 readers -
  • robots.txt: the ultimate guide

    … this the crawl budget. By blocking sections of your site from the search engine spider, you allow your crawl budget to be used for other sections. Especially on sites where a lot of SEO clean up has to be done, it can be very beneficial to first quickly block the search engines from crawling a few sections. blocking query parameters One situation where…

    Joost De Valk/ Yoastin Paid Search- 17 readers -