Skip to content

SercanKaraoglu/log-collector

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

17 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Log Collector

This application broadcast lines from log file. Deployment can be made async, which means log reader or log consumer can be deployed independently because of service discovery feature.

Build Image

While this will build docker container it also produces distribution .zip, .tar and start scripts as well under the build directory.

$ gradle distDocker

###AWS S3 Integration This project have S3 integration if you want to use it then aws profile need to be defined. If we pass docker environment variables like this; -e profile=s3 -e region=us-east-1 -e path=sercan-deneme Then we need to define s3 profile under the ~/.aws/credentials like following

[s3]
aws_access_key_id = your aws id 
aws_secret_access_key = your aws access key

Files of Interest

Each local file reader instance expose whatever file under the path of /data , while S3 file readers expose files under the given path with environment variable path, in the below case it is sercan-deneme. However, only those files that given with -e files parameter will be consumed by the application, in the below clustered case they are first.log, second.log and cloud.log

Clustered Run

Example clustered configuration includes 4 JVM; runs two local path server and one s3 path server and one zookeeper for service discovery. Local path servers serves ${projectDir}/data/first and ${projectDir}/data/second.

$ cd app/clustered
$ docker-compose up

Non Clustered (Single JVM) Run

$ cd app/nonclustered
$ bash run.sh

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published