Crawling And Indexing

  • Closing down for a day

    …, the simplest approach is to disable that specific functionality. In most cases, shopping cart pages can either be blocked from crawling through the robots.txt file, or blocked from indexing with a robots meta tag. Since search engines either won't see or index that content, you can communicate this to users in an appropriate way. For example, you may disable…

    Google Webmaster Central Blog- 15 readers -
  • An update on Google's feature phone crawling & indexing

    … Limited mobile devices, "feature-phones", require a special form of markup or a transcoder for web content. Most websites don't provide feature-phone-compatible content in WAP/WML any more. Given these developments, we've made changes in how we crawl feature-phone content (note: these changes don't affect smartphone content): 1. We've retired…

    Google Webmaster Central Blog- 17 readers -
  • Deprecating our AJAX crawling scheme

    … tl;dr: We are no longer recommending the AJAX crawling proposal we made back in 2009. In 2009, we made a proposal to make AJAX pages crawlable. Back then, our systems were not able to render and understand pages that use JavaScript to present content to users. Because "crawlers … [were] not able to see any content … created dynamically," we…

    Google Webmaster Central Blog- 14 readers -
  • The four steps to appiness

    … of clicks on app deep links jump by 10x. We’ve gotten a lot of feedback from developers and seen a lot of implementations gone right and others that were good learning experiences since we opened up App Indexing back in June. We’d like to share with you four key steps to monitor app performance and drive user engagement: 1. Give your app developer…

    Google Webmaster Central Blog- 12 readers -
  • Best practices for XML sitemaps & RSS/Atom feeds

    … by Googlebot. A common mistake is including URLs disallowed by robots.txt - which cannot be fetched by Googlebot, or including URLs of pages that don't exist. Only include canonical URLs. A common mistake is to include URLs of duplicate pages. This increases the load on your server without improving indexing. Last modification time Specify a last…

    Google Webmaster Central Blogin Google- 12 readers -
  • An improved search box within the search results

    … Webmaster level: All Today you’ll see a new and improved sitelinks search box. When shown, it will make it easier for users to reach specific content on your site, directly through your own site-search pages. What’s this search box and when does it appear for my site? When users search for a company by name—for example, [Megadodo Publications…

    Google Webmaster Central Blog- 16 readers -
  • Testing robots.txt files made easier

    … individual URLs can be quite tricky. To make that easier, we're now announcing an updated robots.txt testing tool in Webmaster Tools. You can find the updated testing tool in Webmaster Tools within the Crawl section: Here you'll see the current robots.txt file, and can test new URLs to see whether they're disallowed for crawling. To guide your way…

    Google Webmaster Central Blog- 24 readers -
  • Android app indexing is now open for everyone!

    … it. As a site owner, you can show your users the right content at the right time — by connecting pages of your website to the relevant parts of your app you control when your users are directed to your app and when they go to your website. Hundreds of apps have already implemented app indexing. This week at Google I/O, we’re announcing a set of new…

    Google Webmaster Central Blog- 27 readers -
  • Directing smartphone users to the page they actually wanted

    … Webmaster level: all Have you ever used Google Search on your smartphone and clicked on a promising-looking result, only to end up on the mobile site’s homepage, with no idea why the page you were hoping to see vanished? This is such a common annoyance that we’ve even seen comics about it. Usually this happens because the website is not properly…

    Google Webmaster Central Blog- 15 readers -
  • Rendering pages with Fetch as Google

    …. After submitting a URL with "Fetch and render," wait for it to be processed (this might take a moment for some pages). Once it's ready, just click on the response row to see the results. Handling resources blocked by robots.txt Googlebot follows the robots.txt directives for all files that it fetches. If you are disallowing crawling of some…

    Google Webmaster Central Blogin Google- 21 readers -
  • Creating the Right Homepage for your International Users

    …. If you implement this scenario on your international site, remember to use the x-default rel-alternate-hreflang annotation for the country selector page, which was specifically created for these kinds of pages. The x-default value helps us recognize pages that are not specific to one language or region. Automatically redirect users…

    Google Webmaster Central Blog- 8 readers -
  • App Indexing updates

    … Webmaster Level: Advanced In October, we announced guidelines for App Indexing for deep linking directly from Google Search results to your Android app. Thanks to all of you that have expressed interest. We’ve just enabled 20+ additional applications that users will soon see app deep links for in Search Results, and starting today we’re making…

    Google Webmaster Central Blog- 5 readers -
  • More Precise Index Status Data for Your Site Variations

    … protocol and hostname. We hope that you’ll find this update useful, and that it’ll help you monitor, identify and fix indexing problems with your website. You can find additional details in our Index Status Help Center article. As usual, if you have any questions, don’t hesitate to ask in our webmaster Help Forum. Posted by Zineb Ait Bahajji, WTA, thanks to the Webmaster Tools team. …

    Google Webmaster Central Blog- 4 readers -
  • Infinite scroll search-friendly recommendations

    … isn’t -- the right example would cause crawling and indexing of duplicative content.</i></div><br><li>Structure URLs for infinite scroll search engine processing. <ul><li>Each component page contains a full URL. We recommend full URLs in this situation to minimize potential for configuration error.<ul><li…

    Maile Ohye/ Google Webmaster Central Blog- 26 readers -