Googlebot is the search bot software used by Google, which collects documents from the web to build a searchable index for the Google Search engine.If a webmaster wishes to restrict the information on their site available to a Googlebot, or another well-behaved spider, they can do so with the appropriate directives in a robots.txt file, or by adding the meta tag to the web page. Googlebot requests to Web servers are identifiable by a user-agent string containing "Googlebot" and a host address containing "".Currently, Googlebot follows HREF links and SRC links. There is increasing evidence Googlebot can execute JavaScript and parse content generated by Ajax calls as well.
Posts about Googlebot
  • What You Need to Make Your Blog Mobile-Friendly

    … analytics data for one of my sites. For example, Tokyoezine (my Japanese travel niche website) gets more than 64% of traffic from search engines. I’ve set the time period from beginning of November 2014 until today. In my Google Analytics dashboard, under the “Audience” menu, I selected “Mobile—>Overview”. This is the screenshot (click to enlarge…

    Erik Emanuelli/ 31 readers -
  • Chilling Effects Blocks Search Engines from Indexing Entire Site

    … of people whose information appears in the database.” It is surprising that Chilling Effects has blocked the entire domain, rather than simply blocking the notices themselves from being indexed. Some pages are technically indexed however. Just how Google will index pages despite a site blocking Googlebot, Google has now included pages in the index…

    Jennifer Slegg/ The SEM Postin Google- 23 readers -
  • Advanced SEO for JavaScript Sites: Snapshot Pages

    … – sending specific content to bots – often a no-no in SEO. This type of cloaking is considered "ethical," however, as the content is pretty much the same as what users would see. Hence the search engines are OK with it. Google has a full AJAX crawling specification that covers the nuts and bolts. The basic idea is one either adds the #! identifier to your…

    Andrew Delamarter/ Search Engine Watchin SEO- 4 readers -
  • New Negative SEO Hurts Google Rankings… Without Using Backlinks

    New Negative SEO Hurts Google Rankings… Without Using Backlinks December 9, 2014 by Jennifer Slegg Leave a Comment Bartosz Góralewicz recently posted a case study where two different sites seem to be impacted by negative SEO. But the big difference with these negative SEO attacks is they were done without the use of any backlinks. Something was seriously wrong with two of my clients’ sites.

    Jennifer Slegg/ The SEM Post- 20 readers -
  • 2015 Planning: What Does Your SEO Strategy Look Like Next Year?

    … site should be technically structured. There are great guides out there, but to touch the highpoints if you’re not using responsive design: Do not block Googlebot from your mobile site On your desktop URLs, add rel=alternate pointing to corresponding mobile URLs On mobile URLs, add rel=canonical pointing to corresponding desktop URLs Create a mobile…

    Erin Everhart/ Search Engine Watchin SEO- 7 readers -
  • Responsive Design Improves SEO

    … seriously. Google Loves Responsive Design As everyone working in SEO treats Google as an almighty deity of the Internet, anything Google says or does, goes! The ultimate gatekeeper of how a website should be enhanced by search engines, optimizers strictly note Google’s rules to ensure that their website is created to the search engine’s preference…

    SEO Nickin SEO- 36 readers -
  • Site Audit: Indexing Tips & Tricks with Screaming Frog [VIDEO]

    … and type in /robots.txt. Not sure what a robots.txt file is? “A robots.txt file is a text file that stops web crawler software, such as Googlebot, from crawling certain pages of your site. The file is essentially a list of commands, such Allow and Disallow, that tell web crawlers which URLs they can or cannot retrieve. So, if a URL is disallowed…

    Tori Cushing/ AuthorityLabs- 49 readers -
  • Optimising Demandware for SEO

    … disallowing both pmin and pmax in robots.txt is recommended. Disallow: /*pmin Disallow: /*pmax Prefn and Prefv1 “prefn” and “prefv” parameters control the majority of the faceted navigation on Sally Beauty. For example if I am on the “eye makeup” page, and select one of the filters, e.g brand (andrea), then I am presented with a prefn/v related URL…

    Sophie Webb/ SEOgadgetin SEO EMail- 50 readers -
Get the top posts daily into your mailbox!
More from around the web