Random Questions

[awick]

Looking thru parsnip it seems very cool!!! I would love to have an independent language that Wireshark/Zeek/Arkime/Suricata/... could all use to generate parsers faster. I've started prototyping one for Arkime.

Some random questions, I'm sure I'll have more.

1. Why snapshot and IL files? What is the difference? Can you have IL files without snapshot files? It looks like maybe you only need snapshot if using the frontend?
2. Would you be open to moving Zeek specific items to having a "zeek" prefix, ex conversionFile => zeekConversionFile, logIndependently => zeekLogIndependently etc? Then we could have arkime* and wireshark* or we could do x-zeek or whatever you want.
3. Is loggingValue Zeek specific (zeekLoggingValue) or should it be renamed description like used for other friendly names
4. signatureFile seems like it could be made generic and only use zeekSignatureFile when it can't be

  signature: {
    ip-protocols: ["tcp"],
    payload: "/^[\x00-\xff]{2}/"
    enable: "PARSER_TCP"
  }

5. Why is config.json in the general dir instead of the top level IL dir? Does it have to be in the general dir?
6. Why have scope: instead of just using a dot notation? Is that to force everything in the "json object" to be the same scope?

{
    "referenceType": "DirectPDUCommand"
    "scope": "general"
}

vs

{
    "referenceType": "general.DirectPDUCommand"
}

or it could still be "referenceType": "DirectPDUCommand" if allowed to assume current scope
7) Why does dependsOn duplicate so much of the type info, and if it doesn't match do I use the dependsOn or the original?
ex:
Hart.DirectPDU

dependsOn: {
    "name": "byteCount",
    "type": "uint",
    "size": 16
}

Hart.DirectPDUCommand

fields: {
    "name": "byteCount",
    "description": "",
    "notes": "",
    "type": "uint",
    "size": 8
},

is it 8 or 16? or probably I just don't understand what is going on.
8) Why doesn't byteCount have a scope above?
9) If I have a tcp/udp protocol where 99% is the same except for the framing (like dns) what is the best way to do that? two config files (tcp_config.json and udp_config.json)? Or is there some variable that tells me which it is have I have 1 entry point.

[piercema]

Here's the answer to the first five questions.

1. The snapshot file is the "save" file from the front-end that can be imported into the front-end to resume editing. It is accurate that you only need snapshot files if you are using the front-end.
2. I would be open to that idea. We will need to evaluate which fields need the update, and get a good idea on the level of effort.
3. loggingValue is only in the enums. It is the human-readable version of the enum value. That choice is ultimately up to you. If you'd prefer to just log the enum value as a number, you can ignore that field. If you'd like to have a human readable option, loggingValue defines it.
4. That's another good idea. We'd be open to that and it definitely fits the goal of abstracting the details from the end user.
5. That's a design choice. It's mainly so that on smaller parsers with a single scope, you don't need the extra parent folder. We'd be open to potentially moving it.

[awick]

7. Ah I found it, its the byteCount from Body which IS a u16.
8. I don't think i can ignore scope if folks create types with the same names in different scopes.
9. How would the switch know if the data is TCP or UDP, is there a special variable? Can you point me to an example of that switch statement? The second part was asking is if instead I should have 2 entry points that lived in 2 config files and everything else was the same. Any kind of example would be useful.

[piercema]

9. We've never encountered that use case. We currently assume that the same parser is used for both TCP and UDP. We may eventually support a separate entry point for TCP and UDP but it will largely be zeek heavy for determining if the instance is TCP or UDP.

[awick]

9. I'm not sure how you could ever have the same parser for UDP and TCP. With UDP the framing is in the udp.length while for tcp you have to have a message length somewhere. Maybe we are talking past each other here? or maybe you are saying the zeek magic code hides the tcp framing and you don't specify that in parsnip?

Instead of magic code, I would rather see
9a tcp vs udp config files (so they could each specify different entry)
9b tcp vs udp entry in same config file like you describe.
9c have 1 entry and have a IL variable "TcpOrUdp" that is set so you could use that in your parsnip to determine how to parse

I guess if you want this to remain zeek specific I should just stop trying to make it something it isn't. :)

[piercema]

9. You are correct that the zeek code obfuscates the TCP and UDP framing and provides just the data to the parser. I'm not sure if you could do something similar in the Arkime parser but as far as I know Zeek takes advantage of the fact that the TCP and UDP frames are always consistent structures. I'd be open to one of the other solutions or a similar solution but I want to brainstorm a little more. The solutions you suggested might interfere with the parsing side. I'll brainstorm and get back to you.

[awick]

9. ok thanks, I guess it really depends where you want zeek to end and parsnip to begin. Another way would be to have the tcp framing "description" also be a separate and more generic config entry when possible, and when not could still be just in spicy. For example tcp dns would be easy to specify in parsnip language. You are right we could also add arkime specific ways.