The Web Robots Pages 
The Web Robots Pages
Web Robots are programs that traverse the Web automatically. Some people call them Web Wanderers, Crawlers, or Spiders. These pages have further information about these Web Robots.
| The Web Robots FAQ | Frequently Asked Questions about Web Robots, from Web users, Web authors, and Robot implementors. |
| Robots Exclusion | Find out what you can do to direct robots that visit your Web site. |
| A List of Robots | A database of currently known robots, with descriptions and contact details. |
| The Robots Mailing List | An archived mailing list for discussion of technical aspects of designing, building, and operating Web Robots. |
| Articles and Papers | Background reading for people interested in Web Robots |
| Related Sites | Some references to other sites that concern Web Robots. |
没有评论:
发表评论