Google pushes for an official web crawler standard – Engadget

1. Google pushes for an official web crawler standard  Engadget
2. Google wants to make robots.txt an Internet standard after 25 years  9to5Google
3. Google Wants to Establish an Official Standard for Using Robots.txt  Search Engine Journal
4. Google open-sources robots.txt parser in push to make Robots Exclusion Protocol an official standard  VentureBeat
5. Google wants to make the 25-year-old robots.txt protocol an internet standard  The Next Web
6. View full coverage on Google News

Source: Google News Feed