Reading List -- CS 6913

Following is the list of papers that are covered or that you have to read as part of this course. Links are provided. You can also easily find electronic versions of most of these papers yourself by pasting the titles into Google.

Part I: Basics of HTTP, Crawling, and Search Engines:

L. Garshol: How the Web Works: HTTP and CGI Explained
A. Rappaport: Robots & Spiders & Crawlers White Paper, Inktomi 1999.
Brin/Page: The Anatomy of a Large-Scale Hypertextual Web Search Engine 7th World Wide Web Conference, 1998
Arasu/Cho/Garcia-Molina/Paepcke/Raghavan: Searching the Web ACM Transactions on Internet Technologies, 2001

Part II: Historical Perspective on Information Science and Retrieval - for your interest:

Vannevar Bush: As We May Think. Atlantic Monthly, 1945.
Michael Buckland: Emanuel Goldberg, Electronic Document Retrieval, And Vannevar Bush's Memex. Journal of the American Society for Information Science, 1992.
Alex Wright: The Web Time Forgot. New York Times, June 17, 2008.
Boyd Rayward: Alle Kennis van de Wereld (Biography of Paul Otlet). Short online documentary movie, 1998.

Additional material -- if you are really interested:

Francoise Levie: The Main Who Wanted to Classify the World. Documentary movie, 2007. (article, movie itself not online)
Alex Wright: Forgotten Forefather: Paul Otlet. Boxes and Arrows 2003.
Lionel Casson: Libraries in the Ancient World. Yale University Press, 2002.
Chris Andersen: The Long Tail. Wired Magazine, October 2004.

Part III: Indexing and Compression

Crochemore/Lecroq: Text Data Compression Algorithms. From: Handbook of Algorithms and Theory of Computation, CRC Press.
Bentley/McIlroy: Data Compression Using Long Common Strings. IEEE Data Compression Conference, 1999.
Sections 1 to 8 of : Zobel/Moffat: Inverted Files for Text Search Engines. Computing Surveys, Volume 38, 2006. (Please retrieve this paper from the ACM Digital Library via the Poly library site.)

Recommended but not required additional reading:

Zhang/Long/Suel: Performance of Compressed Inverted List Caching in Search Engines. 17th International WWW Conference, 2008.
Melnik/Raghavan/Yang/Garcia-Molina: Building a Distributed Full-Text Index for the Web. ACM Transactions on Internet Technologies, 2001.

Part IV: Link-Based Ranking and Mining of the Web Graph

J. Kleinberg: Authoritative Sources in a Hyperlinked Environment. Journal of the ACM, September 1999.
Chakrabarti/Dom/Kumar/et al.: Mining the Web's Link Structure. IEEE Computer, August 1999.
Broder/Kumar/Maghoul/et al.: Graph Structure in the Web. WWW Conference 2000.
Randall/Stata/Wickremesinghe/Wiener: The Link Database: Fast Access to Graphs of the Web. Data Compression Conference, 2002.

Recommended but not required additional reading:

T. Haveliwala: Topic-Sensitive Pagerank. WWW Conference, 2002.
Trawling the Web for Emerging Cyber-Communities. WWW Conference, 1999.
Kumar/Raghavan/Rajagopalan/Tomkins: Extracting Large-Scale Knowledge Bases From the Web. VLDB Conference, 1999.
Boldi/Vigna: The Web Graph Framework I: Compression Techniques. WWW 2004.

Part V: Advanced crawling, web surveillance, and specialized search tools

Lee/Leonard/Wang/Loguinov: IRLbot: Scaling to 6 Billion Pages and Beyond. WWW Conference, 2008.
Cho/Garcia-Molina: The Evolution of the Web and Implications for an Incremental Crawler. Int. Conference on Very Large Data Bases (VLDB), 2000.
Chakrabarti/van den Berg/Dom: Focused Crawling: A New Approach to Topic-Specific Resource Discovery. WWW Conference, 1998.
Broder/Mitzenmacher: Network Applications of Bloom Filters: A Survey. Internet Mathematics Vol. 1, No. 4: 485-509. 2002. (read at least first 5 pages)

Recommended but not required additional reading:

Olston/Pandey: Recrawl Scheduling Based on Information Longevity. WWW Conference, 2008.
Pandey/Olston: Crawl Ordering by Search Impact. ACM WSDM Conference, 2008.
Chakrabarti/Punera/Subramanyam: Accelerated Focused Crawling Through Online Relevance Feedback. WWW 2002, Hawaii.
Marc Najork and Allan Heydon: High-Performance Web Crawling.. SRC Research Report 173, Compaq Systems Research Center (2001).
Shkapenyuk/Suel: Design and Implementation of a High-Performance Distributed Web Crawler. Int. Conference on Data Engineering, February 2002.
Boyapati/Chevrier/Finkel/et al: ChangeDetector: A Site-Level Monitoring Tool for the WWW. WWW Conference, 2002.
Lawrence/Bollacker/Giles: Indexing and Retrieval of Scientific Literature. Conference on Information and Knowledge Management (CIKM), 1999.

Part VI: Search engine architecture, scalable data mining platforms, advanced query processing

Barroso/Dean/Hoelzle: Web Search for a Planet: The Google Cluster Architecture. IEEE Micro, March 2003.
Ghemawat/Gobioff/Leung: The Google File System. SOSP Conference, 2003.
Dean/Ghemawat: MapReduce: Simplified Data Processing on Large Clusters. Sixth Symposium on Operating System Design and Implementation, 2004.
Fagin: Combining Fuzzy Information: an Overview. ACM SIGMOD Record, 2002.

Further optional reading:

Jeff Dean: Challenges in Building Large-Scale Information Retrieval Systems (Video) Talk at WSDM 2009 Conference.
Eric Brewer: Lessons from Giant-Scale Services. IEEE Internet Computing. Vol. 5, No. 4. pp. 46-55. July/August 2001.
Risvik/Michelsen: Search Engines and Web Dynamics. Computer Networks, Volume 39, 2002.
Chang/Dean/et al.: BigTable: A Distributed Storage System for Structured Data. OSDI'06: 7th Symposium on Operating System Design and Implementation, 2006.
Isard/Budiu/Yuan/Birrell/Fetterly: Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks. EuroSys 2007.
Long/Suel: Three-Level Caching for Efficient Query Processing in Large Web Search Engines. 14th International World Wide Web Conference, 2005.

Various Videos of Talks and Other Resources for Additional Information