Skip to content

koolakoff/log_parse

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 

Repository files navigation

A demo project of logs parser tool

contain 
- parse_lib.py - engine for parsing specified events
- parse_usb.py - an example of usage - a use case for tracking USB stick event detection from dmesg logs
- log_example_1_usb - the piece of dmesg report used as a log example


Example usage:
- insert a USB stick to Linux PC (tested only with Transcend mass storage USB stick)
- get the kernel logs to separate file and apply parser over the file:
   $ dmesg > log
   $ ./parse_usb.py log
   log_example_1_usb +17  [  199.309030] USB storage detected: sda1

The parse engine supports combination of easy-to-use API and flexibility.
Parser engine supports
- single regexp
- external functions for parsing specific use cases
- events represented with multiple logs lines

Usage:
to use the engine, just need register set of Parser objects and then feed the logs lines to each of these objects

parser = Parser(key, [report_line])
 - where 'key' defines what would be parsed (see details below)
 - report_line - the actual text to display if event found in the log


the 'key' supports the following use cases:
- track one line log. In scope of this use case it is possible to register 'key' as a
  -- string regexp - that would be chedked by logs parser engine automatically
       Example:
          parser_usb = Parser(r"usb.*Product:\ Mass\ Storage\ Device", report_line="USB detected")
       Note, when passing a line as a regexp use letter r in front of string like this
         r"some\ regexp\ line"
         the 'r' tells python to store all '\' as a symbol of '\' but not as a hash-combo
       Note, it is used re.VERBOSE regexp style
  -- external function handler - the parser engine will call this handler for log analyze
     this allow parse more detailed info (i.e. when needs parse some info from log line)
       Example:
          def parse_hello(line):
             if "hello" in line: return "a Hello event"
          parser_usb = Parser(parse_hello)
- track multiple lines log. 
  some events may be represented in logs with multiple lines.
  For example the USB storage detection in dmesg are represented with a set of lines that
  include report of USB mass storage detection, and the corresponding device name
  is reported on a separate line.
  in this case it is possible to register the list of handlers (regex or external functions) 
  for each of line. The parser will print result only when each of lines from the set is catched
  Example:
     def parse_world(line):
        if "world" in line: return "a Hello world event"
     parser_usb = Parser([r"hello",parse_world])

About

a Python demo tool of logs parser

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages