Distributed Search Engine

A search engine partially based on Google, circa 1998.

• Retrieval and ranking incorporates BM25 and PageRank
• Distributed crawler/indexer/link analysis for computing document index and metadata
• A RESTful server that supports server-side caching and concurrent queries

See our technical report for system design, scalability, and more search demos.

Extra Features

• Excerpts with highlighted hits are loaded dynamically and shown on the result page.
• Web UI integrates search results from News and Yelp webservices.
• Web UI supports search query autocomplete.
• Web UI supports loading 10 pages of search results.

Tech

• ReactJS
• MUI
• News API
• Yelp Fushion API
• IP Geolocation API
• Spark Java
• JDBC
• HikariCP
• Jsoup
• Guava
• OpenNLP

Quick Start

Server

Specify server/src/main/resources/config.properties that contains your API keys and database credentials (not provided).

db.url=jdbc:postgresql://host:port/database
db.user=username
db.pass=password
news.apiKey=abcdefghijk
yelp.apiKey=abcdefghijk

cd server
mvn clean install
mvn exec:java

Client

To run on local development machine, node >= 14.0.0 is required.

cd client
npm i
npm start

Precomputed Components

See READMEs below for implemented features, source files, and instructions of each component.

• Indexer
• PageRank
• Crawler