A log ingestor system that can efficiently handle vast volumes of log data and a query interface that can be used for posting and filtering the queries. Both the programs have been made using Javascript.
- Node.js for the backend
- Elastic Search for querying
- ReactJS for the frontend query interface
- Docker for containerization
To ensure scalility and efficiency of the backend system, Elastic Search has been used that uses indexing and sharding to efficiently search through huge volumes of data.
A query can be posted either through the web interface or a POST request can be made to http://localhost:3000/.
- Filters made based on all the parameters present in the request body
- For the message filter, a search can be made using any substring within the message body. For the other parameters, the search word has to be exact as those are unique fields/keywords
- Search within specific timestamps has been implemented
- Multiple filters can be combined for results
- Real time log ingestion and searching can be done
- Stateful logs - The logs remain stored even after the process is killed. If this is not required, this line
docker-compose downcan be uncommented in thescript.shfile in the root directory
Run .\script.sh in the root directory.
The server and client automatically start after elastic search is in healthy state.
- Role-based access to the query interface can be implemented
- Utilizing regex for searching