Skip to content

Latest commit

 

History

History
114 lines (87 loc) · 4.97 KB

README.md

File metadata and controls

114 lines (87 loc) · 4.97 KB
Raw

Ginx

Ginx is a fast Nginx log parser written in Javascript; It can persist cursors of each file, to continue where it left off in case of a shutdown, unexpected exception or Ctrl+D, all that with the option of parsing a directory of files instead of one file.

EXAMPLE USAGE

var Ginx = require("ginx");
var ginx = new Ginx();
//example read from file
ginx.parseFile("nginx_prod.log",
  function(err, row){
	  if (err) throw err;
	  console.log("this will print each parsed line:" + JSON.stringify(row));
  },
  function(err, file){
	if (err) throw err;
	 console.log(file + " parsing complete");
  }
);

API

Ginx

##Ginx(format, options={})##

Construct a 'new Ginx();

  • Arguments
    • format - Optional string representing the Nginx access_log format, usually in your nginx conf, check ./lib/ginx.js source to see the default format if you don't pass any.
    • options - Optional hash that may contain two key:value pairs - default {'persistent': true, 'fieldsToObjects': false}
      • 'persistent' defaults to true, whether your program will persist file positions to a local file in ./tmp/stored.cursors
      • 'fieldsToObjects' defaults to false, whether the program will attempt to parse every column to its corresponding object, i.e Date, Number, or Null - if turned to True, it may impact the performance depending on Number and Size of files getting parsed.
      • 'storageFile' defaults to ./tmp/stored.cursors, which file you want to store the cursors in
        • 'originalText' - defaults to false, a boolean, if true, it will augment each row JSON object with its original text on an __originalText property

##ginx.parseLine(line, rowCallback, options={})##

Parse one line string

  • Arguments
    • line - a string representing the line to be parsed
    • callback(err, row) - a callback function with the error if any, and the result row object which may contain custom attributes parsed from the format Also, the result object has more attributes __file which is the file parsed from, __originalText which the original text before parsing and __lastrow if it is
    • options, optional hash, currently only supports 'file': path

##ginx.parseFile(path, rowCallback, fileCallback, options={})##

Parse a file

  • Arguments
    • path - a string representing the path to the file to be parsed
    • rowCallback(err, row) - a callback function for each row's end of parse, with the error if any, and the result row object
    • fileCallback(err, file) - a callback function for file's end of parse
    • dirCallback(err) - an Optional callback function for Dir's end of parse, if any
    • options, optional hash, currently only supports 'isPool': boolean, which is used internally to determine if this file was in the waiting pool or not, you don't need it at all if you're parsing a single file, even if you're not, parseDir will take care of that part if you're parsing multiple files.

##ginx.parseDir(directory, rowCallback, fileCallback, dirCallback)##

Parse all files in the first level of a directory (they have to all have the same format)

  • Arguments
    • directory - a string representing the directory that has the files to be parsed
    • rowCallback(err, row) - a callback function for each row's end of parse, with the error if any, and the result row object
    • fileCallback(err, file) - a callback function for file's end of parse
    • dirCallback(err, filesParsedCount) - a callback function for Dir's end of parse

##ginx.pause([readFile], [callback])##

Pauses all ReadStreams in Ginx Memory, or a single file's stream if passed.

  • Arguments
    • readFile - optional, the file path of the file being read by the ReadStream, if not passed, it will try to pause them all
    • callback - optional, the callback after pausing completes

##ginx.resume([readFile], [callback])##

Resumes all ReadStreams in Ginx Memory, or a single file's stream if passed.

  • Arguments
    • readFile - optional, the file path of the file being read by the ReadStream, if not passed, it will try to resume them all.
    • callback - optional, the callback after resuming completes

##ginx.stop([callback])##

Stops all ReadStreams in Ginx Memory, then persists all cursors in memory

  • Arguments
    • callback - optional, the callback after stoping completes

INSTALL

	npm install ginx

or manually git clone https://github.com/akhoury/ginx.git # cp the ginx dir into your program's node_modules npm install # to install dependencies

you can 'node demo.js' to try it out quick, but please open demo.js and read the comments quickly.

##TODO##

  • add tail -f feature to the command line tool using fs.watch or node-tail module
  • add DeleteStorage to the API
  • add auto detection feature for the format based on an example log line.
  • add support for Error logs
  • log with different levels (info, error, warning, debug, trace) to an optional log file
  • refactor some of the big functions
  • support other encodings
  • optional DEBUG param flag, and input/output paths flags
  • increase test coverage, seriously ... along with few others entered as issues.