Crawl Directives

    • SEO for a new website: the very first things to do

      How does a new website start ranking? Does it just magically appear in Google after you’ve launched it? What things do you have to do to start ranking in Google and get traffic from the search engines? Here, I explain the first steps you’ll need to take right after the launch of your new website. Learn how to start working on the SEO for a new website! First: you’ll need to hav ...

      Marieke Van De Rakt/ Yoastin SEO- 18 readers -
  • Preventing your site from being indexed, the right way

    … prevents indexing of your site. Let me explain how this works in this post. For more on robots.txt, please read robots.txt: the ultimate guide. Become a technical SEO expert with our Technical SEO 1 training! » $ 199€ 199 - Buy now » Info There is a difference between being indexed and being listed in Google Before we explain things any further, we…

    Joost De Valk/ Yoastin SEO- 12 readers -
  • Ask Yoast: Block your site’s search results pages?

    … Every website should have a decent internal search functionality that shows the visitors search results that fit their search query. However, those search results pages on your site don’t need to be shown in Google’s search results. In fact, Google advises against this too; it’s not a great user experience to click on a Google search result, just…

    Joost De Valk/ Yoast- 15 readers -
  • Block your site’s search result pages

    … content management systems allow for templates for your site’s search results as well, so adding a simple line of code to that template will suffice: Become a technical SEO expert with our Technical SEO 1 training! » $ 199€ 199 - Buy now » Info Meta robots AND robots.txt? If you try to block internal search pages by adding that meta robots tag…

    Michiel Heijmans/ Yoastin SEO- 16 readers -
  • SEO basics: What is crawlability?

    … of the most important concepts of technical SEO: crawlability. What is the crawler again? A search engine like Google consists of a crawler, an index and an algorithm. The crawler follows the links. When Google’s crawler finds your website, it’ll read it and its content is saved in the index. A crawler follows the links on the web. A crawler is also…

    Marieke Van De Rakt/ Yoastin SEO- 14 readers -
  • Ask Yoast: should I redirect my affiliate links?

    … There are several reasons for cloaking or redirecting affiliate links. For instance, it’s easier to work with affiliate links when you redirect them, plus you can make them look prettier. But do you know how to cloak affiliate links? We explained how the process works in one of our previous posts. This Ask Yoast is about the method of cloaking…

    Joost De Valk/ Yoastin Affiliate- 20 readers -
  • Ask Yoast: nofollow layered navigation links?

    … of this?” Check out the video or read the answer below! https://www.youtube.com/watch?v=v9NPmgg7p6M Want to outrank your competitor and get more sales? Read our Shop SEO eBook! » $ 25€ 25 - Buy now » Info Layered navigation links Read this transcript to learn how to deal with layered or faceted navigation links: “The question is: “Why would you…

    Joost De Valk/ Yoastin SEO- 21 readers -
  • How to cloak your affiliate links

    … We regularly consult for sites that monetize, in part, with affiliate links. We usually advise people to redirect affiliate links. In the past, we noticed that there wasn’t a proper script available online that could handle this for us, so we created one to tackle this problem. In this post, I explain how you can get your hands on it and how you…

    Joost De Valk/ Yoastin Affiliate SEO How To's- 41 readers -
  • Playing with the X-Robots-Tag HTTP header

    … different kinds of directives: crawler directives and indexer directives. I’ll briefly explain the difference below. Get the most out of Yoast SEO, learn every feature and best practice in our Yoast SEO for WordPress training! » $ 99€ 99 - Buy now » Info Crawler directives The robots.txt file only contains the so called ‘crawler directives’, which tells…

    Joost De Valk/ Yoastin SEO- 28 readers -
  • Crawl budget optimization

    … Google doesn’t always spider every page on a site instantly. In fact, sometimes it can take weeks. This might get in the way of your SEO efforts. Your newly optimized landing page might not get indexed. At that point, it becomes time to optimize your crawl budget. Crawl budget is the time Google has in a given period to crawl your site. It might…

    Joost De Valk/ Yoast- 22 readers -
  • robots.txt: the ultimate guide

    … this the crawl budget. By blocking sections of your site from the search engine spider, you allow your crawl budget to be used for other sections. Especially on sites where a lot of SEO clean up has to be done, it can be very beneficial to first quickly block the search engines from crawling a few sections. blocking query parameters One situation where…

    Joost De Valk/ Yoastin Paid Search- 17 readers -
  • rel=canonical • What it is and how (not) to use it

    … In February 2009, six years to the day from when this is published, Google, Bing and Yahoo! introduced the rel=canonical link element (Matt’s post is probably the easiest reading). While the idea is simple, the specifics of how to use it turn out to be complex. The basic premise is: if you have several similar versions of the same content, you…

    Joost De Valk/ Yoastin Social How To's- 23 readers -