Skip to content

Latest commit

 

History

History
283 lines (240 loc) · 8.76 KB

FAQ.md

File metadata and controls

283 lines (240 loc) · 8.76 KB
Raw

FAQ and Examples

What output-options are there?

You can get XML, CSV, JSON or TABLE (CSV with spaces).

Does PacketQ conform to the SQL standard ?

The short answer is no.

Packet implements an SQL like select function with some notable omissions like:

  • No 3 value logic i.e. theres no special NULL value
  • No supports for joins or subqueries
  • No support for the distinct keyword
  • No support for like in where statements

Refer to the file grammar for supported statements.

How do I return full packetdata for the first 3 dns packets?

Use this SQL-statement.

$ packetq -s "select * from dns limit 3" packets.pcap

How do I check which qname is queried the most?

$ packetq -s "select qname,count(*) as count from dns group by qname order by count desc limit 1 " packets.pcap
[
{ "table_name": "result",
    "head": [
      { "name": "qname","type": "text" },
      { "name": "count","type": "int" }
    ],
    "data": [ ["se.",2747]  ]
  }
]

How do I check which server that makes the most queries?

$ packetq -s "select src_addr,count(*) as count from dns group by src_addr order by count desc limit 1" packets.pcap
[
  {
    "table_name": "result",
    "head": [
      { "name": "src_addr","type": "text" },
      { "name": "count","type": "int" }
    ],
    "data": [
["::127.0.0.1",1849]
    ]
  }
]

I have 90000000 packets in a file, how can I speed up the query?

By using the sample-keyword before the actual query, it selects only every Xth packet and speeds up the query substantially. The amount of time saved by this command will be different on different systems, at some point the time it takes to read the file from disk will be longer than the time spent on processing. In other words, the full file will have to be read from disk regardless. This command also saved RAM.

$ ./packetq -s "sample 1000; select count(*) from dns" ~/pcap/*
[  {
    "table_name": "result",
    "head": [ { "name": "count(1)","type": "int" }],
    "data": [ [90000]] }]

How do I extract parts of the qname and group using this, like the domain-name?

Using the rsplit-function you can split values based on chars (like the dot). This query extract the two lowest levels (the domain for the .se-zone) and makes a top-10 list.

$ packetq -s "select count(*) as Count, lower(rsplit(qname,2)||'.'||rsplit(qname,1)) as Domain from dns group by domain order by count desc limit 10" peak/07/G.ns.se-20110408-074500-em0.gz
[
  {
    "table_name": "result",
    "head": [
      { "name": "Count","type": "int" },
      { "name": "Domain","type": "text" }
    ],
    "data": [
[5130,"pp.se"],
[1686,"netnod.se"],
[1448,".se"],
[1318,"domainnetwork.se"],
[936,"ballou.se"],
[867,"loopia.se"],
[784,"ns.se"],
[736,"sonera.se"],
[694,"digidns.se"],
[639,"prq.se"]
    ]
  }
]

How do I make a top-list of client-IPs?

PacketQ supports looking at packets to and from the nameserver/resolver. But since the client-IP changes from src_addr to dst_addr if the packet is an reply, we have to use the if-function to extract the client-IP.

$ packetq -s "select count(*) as Count, if(qr=1,dst_addr,src_addr) as Source from dns group by source order by count desc limit 15" peak/07/G.ns.se-20110408-070000-em0.gz
[
  {
    "table_name": "result",
    "head": [
      { "name": "Count","type": "int" },
      { "name": "Source","type": "text" }
    ],
    "data": [
[1021,"202.67.195.41"],
[929,"91.191.143.34"],
[638,"195.84.206.2"],
[582,"202.101.173.146"],
[490,"2a02:aa0:0:10:214:4fff:fef2:61a0"],
[438,"80.251.201.177"],
[436,"192.89.123.231"],
[426,"81.228.8.143"],
[409,"81.228.8.69"],
[402,"65.54.237.136"],
[368,"81.228.8.144"],
[352,"213.50.29.170"],
[342,"217.28.204.2"],
[318,"2001:6f0:0:1::2:3"],
[297,"81.228.9.132"]
    ]
  }
]

Does PacketQ support IPv6?

Yes, it does. See the above result.

Can I make a query that lists all resolvers that are broken?

If by broken you mean resolvers that are repeating the same query over and over again, then yes.

$ packetq -s "select count(*) as Count,qname,if(qr=1,dst_addr,src_addr) as Source,qtype from dns group by qname,source,qtype order by count desc limit 10" peak/07/G.ns.se-20110408-070000-em0.gz
[
  {
    "table_name": "result",
    "head": [
      { "name": "Count","type": "int" },
      { "name": "qname","type": "text" },
      { "name": "Source","type": "text" },
      { "name": "qtype","type": "int" }
    ],
    "data": [
[607,"se.","195.84.206.2",48],
[375,"ns.digidns.se.","202.67.195.41",1],
[330,"sas.sas.se.","202.67.195.41",1],
[316,"NS.DIGIDNS.SE.","202.67.195.41",1],
[147,"se.","62.80.200.144",48],
[42,"se.","94.232.104.58",48],
[40,"_ldap._tcp.pdc._msdcs.telemgmt.se.","213.115.146.180",33],
[40,"se.","83.12.96.122",48],
[34,"www.membran.se.","217.73.15.38",1],
[28,"fc.edu.upplandsvasby.se.","193.12.6.247",1]
    ]
  }
]

Can I get a list of what qtypes are used the most?

Yes, and PacketQ can even translate the headers into text for you by using the NAME-function.

$ packetq -s "SELECT NAME( 'qtype' , qtype ) AS qt, COUNT(*) AS antal FROM dns GROUP BY qtype ORDER BY Antal DESC" peak/07/G.ns.se-20110408-070000-em0.gz
[
  {
    "table_name": "result",
    "head": [
      { "name": "qt","type": "text" },
      { "name": "antal","type": "int" }
    ],
    "data": [
["A",59908],
["AAAA",13076],
["MX",10527],
["DS",7545],
["NS",1666],
["DNSKEY",1077],
["TXT",969],
["A6",563],
["*",556],
["SRV",197],
["SPF",145],
["SOA",126],
["PTR",60],
["CNAME",47],
["NAPTR",3]
    ]
  }
]

OK, give me a more advanced query to chew on?

Absolutely! This query uses the TRUNCATE-keyword to split the stats into different timeperiods. In this case 10-second periods. This is great for line-graphs that you can make using Excel or similar tools.

We then get the stats by using conditions like ((rcode=0) and (an_count=0)) to identify packets that resulted in referrals and (rcode=3) to see which packets got an nxdomain back and so on. By using the SUM-function they are all counted and since we want q/sec as our output, divided by 10.

Try to import the result below into Excel!

$ packetq --table -s "select s as timestamp, count(*)/10 as total, sum((rcode=0) and (an_count=0))/10 as referral, sum(rcode=3)/10 as nxdomain, sum((rcode=0) and (an_count>0))/10 as success, sum(rd)/10 as recursion, sum(ether_type=34525)/10 as IPV6, sum(protocol=6)/10 as TCP from dns group by truncate(s/10) " peak/07/G.ns.se-20110408-071000-em0.gz
"timestamp","total","referral","nxdomain","success","recursion","IPV6","TCP"
1302246609 ,822.5  ,304.3     ,515.5     ,2        ,527        ,15.3  ,0
1302246619 ,1135.9 ,330.8     ,797.8     ,6        ,817.2      ,16.7  ,0
1302246629 ,1475.2 ,317.8     ,1152.1    ,4.4      ,1170       ,17.9  ,0
1302246639 ,1774.7 ,344.9     ,1424.8    ,3.8      ,1461.4     ,14.2  ,0
1302246649 ,2014.4 ,350.6     ,1658.5    ,4.4      ,1712.4     ,15.9  ,0
1302246659 ,2136.4 ,346.7     ,1786.2    ,2.5      ,1834.9     ,15.4  ,0
1302246669 ,2212.1 ,336       ,1872      ,3        ,1918.3     ,16.1  ,0.1
1302246679 ,2292.4 ,336.7     ,1952.6    ,2.1      ,1999.7     ,16.4  ,0
1302246689 ,2337   ,324.2     ,2004.6    ,6.8      ,2052.6     ,13.8  ,0
1302246699 ,2410.8 ,315.9     ,2084.7    ,9.1      ,2121.7     ,12.3  ,0
1302246709 ,2437.5 ,311.9     ,2113      ,12       ,2151.3     ,14.2  ,0
1302246719 ,2488.1 ,302.8     ,2178.5    ,5.7      ,2214.8     ,13.8  ,0
1302246729 ,2496   ,327.4     ,2163.6    ,3.8      ,2202.5     ,15    ,0
1302246739 ,2546.7 ,333.3     ,2200.2    ,11.6     ,2234.9     ,16.7  ,0
1302246749 ,2473.3 ,314.1     ,2150.4    ,7.9      ,2180.8     ,15.6  ,0
1302246759 ,2476.2 ,313.4     ,2146.8    ,14.5     ,2175.3     ,12    ,0
1302246769 ,2472   ,324.9     ,2133.2    ,12.8     ,2159.4     ,14.3  ,0
1302246779 ,2464.8 ,337.6     ,2120.4    ,5.7      ,2156.7     ,14.6  ,0
1302246789 ,2443.9 ,332.4     ,2107.8    ,2.9      ,2141.5     ,14.4  ,0
1302246799 ,2447.6 ,323.6     ,2119.8    ,2.8      ,2161.9     ,14.9  ,0
1302246809 ,2477.9 ,324       ,2150.7    ,1.7      ,2184.2     ,15.1  ,0
1302246819 ,2440.2 ,321.9     ,2114.1    ,3.1      ,2148       ,14.9  ,0.1
1302246829 ,2478   ,312.6     ,2162.4    ,2.4      ,2199.5     ,15    ,0
1302246839 ,2537.3 ,319.1     ,2214.6    ,2.3      ,2260.3     ,16.3  ,0
1302246849 ,2654.5 ,304.2     ,2348.1    ,1.3      ,2382.5     ,15.6  ,0
1302246859 ,2669.8 ,318.6     ,2347.8    ,2.9      ,2381.8     ,15.7  ,0
1302246869 ,2707.6 ,328.2     ,2375.8    ,2.8      ,2414.9     ,15.1  ,0
1302246879 ,2792.7 ,309.3     ,2480.5    ,1.8      ,2501.3     ,16.7  ,0
1302246889 ,2841.3 ,299.7     ,2532.7    ,7.8      ,2559.4     ,16.3  ,0
1302246899 ,2857.9 ,312.8     ,2531.7    ,12.5     ,2564.4     ,16.4  ,0