The Apple Search Bot is Actually a Crawler to Improve Siri Not New Search Engine

by Jennifer Slegg
Everyone was buzzing earlier this week with rumors that Apple was crawling the web. It began popping up in the middle of October with Apple’s IP range with a NetName of APPLE-WWNET as “Mozilla/5.0 (compatible; Fetcher/0.1)” written in Go. The crawler does request robots.txt and is still actively crawling. But the crawler is only taking straight HTML and not the CSS, JavaScript or image files.Read the full article