- Our Blog
In 1994, the Robot Exclusion Protocol was established as a voluntary system for webmasters to control which robots could access which pages of a website. In a nutshell, under the system, robots such as Googlebot announce themselves to Web servers and download a file, robots.txt, that says which pages are allowed and not allowed to be crawled.
Last week’s rising anxiety for digital marketers was the release of Apple’s iOS 9 with ad-blocking technology. In “Welcome to hell: Apple vs. Google vs. Facebook and the slow death of the Web,” Nilay Patel, editor-in-chief of The Verge, worries that ad blocking is going to kill small publishers and leave the Web a fragmented set of proprietary silos controlled by big tech companies.