ISP logs parser

Introduction

Often dishonest ISP gives unreadable (by human beings) logs when asked for details of internet traffic expenses.
This issue is especially actual if you pay by amount of traffic.

isp-logs-parser was developed to help ISP customers extract needed info from such logs.
For instance, to answer the most popular question “which sites I’ve spent all my traffic for?”

Example of unreadable log:

timestamp account_id source destination t_class packets bytes sport dport date
1250878466 144 77.235.9.36 217.29.21.21 260 5 552 51003 80 Sat Aug 22 00:14:26 2009
1250878466 144 217.29.21.21 77.235.9.36 270 4 503 80 51003 Sat Aug 22 00:14:26 2009
1250878469 144 77.235.9.36 217.29.21.21 260 5 534 51005 80 Sat Aug 22 00:14:29 2009
1250878469 144 217.29.21.21 77.235.9.36 270 4 539 80 51005 Sat Aug 22 00:14:29 2009
1250878471 144 91.205.50.34 77.235.9.36 270 7 829 50536 8483 Sat Aug 22 00:14:31 2009
1250878472 144 77.235.9.36 217.29.21.21 260 5 542 51007 80 Sat Aug 22 00:14:32 2009
1250878472 144 217.29.21.21 77.235.9.36 270 4 551 80 51007 Sat Aug 22 00:14:32 2009
1250878472 144 77.235.9.36 217.29.21.21 260 5 536 51008 80 Sat Aug 22 00:14:32 2009
......

Example of isp-logs-parser output:

+-----------------+---------+-----------+----------+
| host            | traffic | day/night | location |
+-----------------+---------+-----------+----------+
| 212.112.103.210 | 35.3GB  | nightly   | internal |
| 81.20.21.134    | 1.94GB  | daily     | internal |
| 91.207.97.23    | 464MB   | nightly   | internal |
| 217.29.28.102   | 337MB   | nightly   | internal |
| 195.38.161.22   | 334MB   | nightly   | internal |
| 217.29.28.102   | 294MB   | daily     | internal |
| 77.95.63.107    | 228MB   | nightly   | internal |
| 86.40.182.30    | 101MB   | nightly   | world    |
| 86.40.182.30    | 54.3MB  | daily     | world    |
| 87.248.210.200  | 36.1MB  | nightly   | world    |
| 87.248.209.186  | 33.4MB  | nightly   | world    |
| 217.29.21.21    | 10.7MB  | daily     | internal |
| 91.205.50.59    | 10.6MB  | daily     | internal |
| 82.99.19.11     | 4.03MB  | nightly   | world    |
| 85.113.0.25     | 3.83MB  | daily     | internal |
| 92.122.213.202  | 1.59MB  | nightly   | world    |
| 212.47.219.83   | 358kB   | daily     | world    |
| 65.54.167.92    | 282kB   | daily     | world    |
| 212.47.219.86   | 260kB   | daily     | world    |
| 207.46.26.185   | 220kB   | daily     | world    |
+-----------------+---------+-----------+----------+

Installation

$ git clone git://github.com/DanielVartanov/isp-logs-parser.git
$ git submodule init
$ git submodule update

Running as a console command

$ parse\!.rb the-log-file

Running as a web-server

$ server.rb
then access it at localhost:4567 by default (yes, Sinatra)

Copyright © 2009 Daniel Vartanov, released under the MIT license