Crawling And Indexing

  • Closing down for a day

    … the search results. You can also include a "Retry after" header to indicate how long the site will be unavailable. Blocking a site for longer than a week can have negative effects on the site's search results regardless of the method that you use. Option: Switch whole website off Turning the server off completely is another option. You might also do…

    Google Webmaster Central Blog- 16 readers -
  • An update on Google's feature phone crawling & indexing

    … the feature-phone Googlebot We won't be using the feature-phone user-agents for crawling for search going forward. 2. Use "handheld" link annotations for dynamic serving of feature-phone content. Some sites provide content for feature-phones through dynamic serving, based on the user's user-agent. To understand this configuration, make sure your…

    Google Webmaster Central Blog- 25 readers -
  • Deprecating our AJAX crawling scheme

    … tl;dr: We are no longer recommending the AJAX crawling proposal we made back in 2009. In 2009, we made a proposal to make AJAX pages crawlable. Back then, our systems were not able to render and understand pages that use JavaScript to present content to users. Because "crawlers … [were] not able to see any content … created dynamically," we…

    Google Webmaster Central Blog- 15 readers -
  • The four steps to appiness

    … Webmaster Level: intermediate to advanced App deep links are the new kid on the block in organic search, and they’re picking up speed faster than you can say “schema.org ViewAction”! For signed-in users, 15% of Google searches on Android now return deep links to apps through App Indexing. And over just the past quarter, we've seen the number…

    Google Webmaster Central Blog- 13 readers -
  • Best practices for XML sitemaps & RSS/Atom feeds

    …, containing only the most recent updates to your site. XML sitemaps are downloaded less frequently than RSS/Atom feeds. For optimal crawling, we recommend using both XML sitemaps and RSS/Atom feeds. XML sitemaps will give Google information about all of the pages on your site. RSS/Atom feeds will provide all updates on your site, helping Google…

    Google Webmaster Central Blogin Google- 14 readers -
  • An improved search box within the search results

    … Webmaster level: All Today you’ll see a new and improved sitelinks search box. When shown, it will make it easier for users to reach specific content on your site, directly through your own site-search pages. What’s this search box and when does it appear for my site? When users search for a company by name—for example, [Megadodo Publications…

    Google Webmaster Central Blog- 19 readers -
  • Testing robots.txt files made easier

    … Webmaster level: intermediate-advanced To crawl, or not to crawl, that is the robots.txt question. Making and maintaining correct robots.txt files can sometimes be difficult. While most sites have it easy (tip: they often don't even need a robots.txt file!), finding the directives within a large robots.txt file that are or were blocking…

    Google Webmaster Central Blog- 27 readers -
  • Android app indexing is now open for everyone!

    … it. As a site owner, you can show your users the right content at the right time — by connecting pages of your website to the relevant parts of your app you control when your users are directed to your app and when they go to your website. Hundreds of apps have already implemented app indexing. This week at Google I/O, we’re announcing a set of new…

    Google Webmaster Central Blog- 28 readers -
  • Directing smartphone users to the page they actually wanted

    … Webmaster level: all Have you ever used Google Search on your smartphone and clicked on a promising-looking result, only to end up on the mobile site’s homepage, with no idea why the page you were hoping to see vanished? This is such a common annoyance that we’ve even seen comics about it. Usually this happens because the website is not properly…

    Google Webmaster Central Blog- 17 readers -
  • Rendering pages with Fetch as Google

    …. After submitting a URL with "Fetch and render," wait for it to be processed (this might take a moment for some pages). Once it's ready, just click on the response row to see the results. Handling resources blocked by robots.txt Googlebot follows the robots.txt directives for all files that it fetches. If you are disallowing crawling of some…

    Google Webmaster Central Blogin Google- 25 readers -
  • Creating the Right Homepage for your International Users

    …. If you implement this scenario on your international site, remember to use the x-default rel-alternate-hreflang annotation for the country selector page, which was specifically created for these kinds of pages. The x-default value helps us recognize pages that are not specific to one language or region. Automatically redirect users…

    Google Webmaster Central Blog- 9 readers -
  • App Indexing updates

    … app deep links to English content available globally. We’re continuing to onboard more publishers in all languages. If you haven’t added deep link support to your Android app or specified these links on your website or in your Sitemaps, please do so and then notify us by filling out this form. Here are some best practices to consider when…

    Google Webmaster Central Blog- 5 readers -
  • More Precise Index Status Data for Your Site Variations

    … Webmaster Level: Intermediate The Google Webmaster Tools Index Status feature reports how many pages on your site are indexed by Google. In the past, we didn’t show index status data for HTTPS websites independently, but rather we’d included everything in the HTTP site’s report. In the last months, we’ve heard from you that you’d like to use…

    Google Webmaster Central Blog- 4 readers -
  • Infinite scroll search-friendly recommendations

    … isn’t -- the right example would cause crawling and indexing of duplicative content.</i></div><br><li>Structure URLs for infinite scroll search engine processing. <ul><li>Each component page contains a full URL. We recommend full URLs in this situation to minimize potential for configuration error.<ul><li…

    Maile Ohye/ Google Webmaster Central Blog- 27 readers -