Crawling And Indexing

  • An update on Google's feature phone crawling & indexing

    … Limited mobile devices, "feature-phones", require a special form of markup or a transcoder for web content. Most websites don't provide feature-phone-compatible content in WAP/WML any more. Given these developments, we've made changes in how we crawl feature-phone content (note: these changes don't affect smartphone content): 1. We've retired…

    Google Webmaster Central Blog- 14 readers -
  • Deprecating our AJAX crawling scheme

    … tl;dr: We are no longer recommending the AJAX crawling proposal we made back in 2009. In 2009, we made a proposal to make AJAX pages crawlable. Back then, our systems were not able to render and understand pages that use JavaScript to present content to users. Because "crawlers … [were] not able to see any content … created dynamically," we…

    Google Webmaster Central Blog- 14 readers -
  • The four steps to appiness

    … of clicks on app deep links jump by 10x. We’ve gotten a lot of feedback from developers and seen a lot of implementations gone right and others that were good learning experiences since we opened up App Indexing back in June. We’d like to share with you four key steps to monitor app performance and drive user engagement: 1. Give your app developer…

    Google Webmaster Central Blog- 10 readers -
  • Best practices for XML sitemaps & RSS/Atom feeds

    …, containing only the most recent updates to your site. XML sitemaps are downloaded less frequently than RSS/Atom feeds. For optimal crawling, we recommend using both XML sitemaps and RSS/Atom feeds. XML sitemaps will give Google information about all of the pages on your site. RSS/Atom feeds will provide all updates on your site, helping Google…

    Google Webmaster Central Blogin Google- 12 readers -
  • Testing robots.txt files made easier

    … individual URLs can be quite tricky. To make that easier, we're now announcing an updated robots.txt testing tool in Webmaster Tools. You can find the updated testing tool in Webmaster Tools within the Crawl section: Here you'll see the current robots.txt file, and can test new URLs to see whether they're disallowed for crawling. To guide your way…

    Google Webmaster Central Blog- 24 readers -
  • Android app indexing is now open for everyone!

    … it. As a site owner, you can show your users the right content at the right time — by connecting pages of your website to the relevant parts of your app you control when your users are directed to your app and when they go to your website. Hundreds of apps have already implemented app indexing. This week at Google I/O, we’re announcing a set of new…

    Google Webmaster Central Blog- 25 readers -
  • Directing smartphone users to the page they actually wanted

    … to act like a smartphone) and see how your site behaves. Simple but effective. :) 2. Check out Webmaster Tools—we’ll send you a message if we detect that any of your site’s pages are redirecting smartphone users to the homepage. We’ll also show you any faulty redirects we detect in the Smartphone Crawl Errors section of Webmaster Tools: 3…

    Google Webmaster Central Blog- 14 readers -
  • Rendering pages with Fetch as Google

    …. After submitting a URL with "Fetch and render," wait for it to be processed (this might take a moment for some pages). Once it's ready, just click on the response row to see the results. Handling resources blocked by robots.txt Googlebot follows the robots.txt directives for all files that it fetches. If you are disallowing crawling of some…

    Google Webmaster Central Blogin Google- 21 readers -
  • Creating the Right Homepage for your International Users

    …. Show them the content that you consider the most appropriate. Whatever configuration you decide to go with, you should make sure all the pages – including country and language selector pages: Have rel-alternate-hreflang annotations. Are accessible for Googlebot's crawling and indexing: do not block the crawling or indexing of your…

    Google Webmaster Central Blog- 7 readers -
  • App Indexing updates

    … app deep links to English content available globally. We’re continuing to onboard more publishers in all languages. If you haven’t added deep link support to your Android app or specified these links on your website or in your Sitemaps, please do so and then notify us by filling out this form. Here are some best practices to consider when…

    Google Webmaster Central Blog- 4 readers -
  • More Precise Index Status Data for Your Site Variations

    … Webmaster Level: Intermediate The Google Webmaster Tools Index Status feature reports how many pages on your site are indexed by Google. In the past, we didn’t show index status data for HTTPS websites independently, but rather we’d included everything in the HTTP site’s report. In the last months, we’ve heard from you that you’d like to use…

    Google Webmaster Central Blog- 3 readers -
  • Infinite scroll search-friendly recommendations

    … isn’t -- the right example would cause crawling and indexing of duplicative content.</i></div><br><li>Structure URLs for infinite scroll search engine processing. <ul><li>Each component page contains a full URL. We recommend full URLs in this situation to minimize potential for configuration error.<ul><li…

    Maile Ohye/ Google Webmaster Central Blog- 26 readers -
Get the top posts daily into your mailbox!