Polybot: A High-Performance Web Crawler

Polybot is a high-performance distributed web crawler developed in the Department of Computer and Information Science at Polytechnic University as part of an academic research project that explores new techniques for searching and analyzing the World Wide Web. Note: we are not in any way connected to the search site www.polybot.com, which came recently (January 2002) to our attention. We also have nothing to do with the virus of the same name!