Reading packet capture files containing packets/heaps for multiple streams

[rtobar]

The SKA Mid folks have produced some data dumps from their correlator prototype in the shape of packet capture files. The data dumps were done on the "any" libpcap device, which is why they ended up using the Linux SLL format (the support we added in #185). These data dumps also capture packets for heaps belonging to multiple streams, with receivers listening on multiple ports.

What we want to achieve is to "replay" these files, reading these heaps and acting as a spead sender, replicating the individual streams that were originally streamed. What I was originally planning to do was to use a receiver stream with a pcap reader to "receive"/read the heaps, then forward those with my senders to their endpoint. However, the pcap receiver currently cannot be constrained to consider only heaps belonging to a particular stream, or sent to a particular port (and there's no way to infer that information from a heap AFAIR).

Given that understanding, I was planning on adding an option to the udp_pcap reader so that it only considers UDP packets for sent to a specific port; that way I could open N receivers each with a pcap reader for each of the ports I'm interested in, then do my heap forwarding. This would be a simple change I believe, but it means I'd have to read the dump N times. An alternative implementation could feed the N receivers using a single pass over the data dump, but that would be more work to get right I believe.

So:

• Is my current understanding correct that what I want to achieve is not possible, or am I missing something?
• What do you think about the proposals stated above, and would you be open to accept patches in that direction? This would bubble up to the python bindings (our senders and receivers are written in python), although I have the intention in the longer term to move our code to C++ now that's a bit more stable.
• I also want to note that I can simply break down the data capture file via tcpdump to isolate each stream into its own capture file, then setup N receivers with one of the sub-files each. This of course requires some minor pre-processing before the files we receive are usable, and means my program will have to read N files, but doesn't require changes in spead2. I might end up going this way in the short term anyway (we'd be looking at trying this out in about 2 months or so), while a more permanent solution is sought.

[bmerry]

Hi Rick. Your understanding is correct. I can see a few approaches to consider, depending on the exact details of what you're trying to do:

1. Take a look at https://github.com/ska-sa/udpreplay, which I wrote specifically for replaying captured streams. It can use ibverbs acceleration and loads all the packets into memory first, so it can be extremely fast. It will also give the most faithful reproduction because it will be replaying packets rather than heaps, and it can also preserve the original timing. There are a few reasons it might not suit:

• Because it loads all the packets into memory first, it doesn't play nice with captures bigger than your RAM.
• It can send the data all to one address, or to the original addresses, but out of the box you can't specify an arbitrary address mapping. You could probably hack something in; the hardest part is just designing the user interface for it. There are also tools for programmatic editing of pcap files that you could use to rewrite the destination addresses.

2. You could split the original pcap file into separate files. I think you can do that just with tcpdump, using -r to read from the original and -w to write to the output, with a filter expression. That would require multiple passes over the original file of course. Possibly there are other pcap-manipulation tools that could efficiently partition a file, or you could write your own using libpcap.

3. You could add a filter option to the udp_pcap reader, and I'd be happy to accept that. Rather than specifically filtering by port I'd suggest allowing an arbitrary filter to pass to pcap_setfilter.

4. Run a single receiver stream, and try to extend spead2 to pass address information from the reader to the stream to the heap object (as you've suggested). There are a couple of issues with this, which is why I've never implemented it despite also having uses for it:

• A single heap could conceivably comprise packets from multiple sources or destinations (KAT-7 worked like this), so in general it's not clear what "the" port is for a heap.
• If the individual streams have conflicting heap IDs, then this will break heap reassembly (unless the heaps are all one packet long). If the heap IDs are separated, then potentially you can use the heap ID instead.
• The stream and reader interfaces are generic and don't assume IPv4 or even IP (e.g. inproc_reader). It's not clear what type the address information should be to be able to hold useful address information for any reader type.
• Per-packet processing is a critical performance path, so any additional packet processing needs to be very very cheap, or at least be very very cheap when not in use.

[rtobar]

Hi Bruce. Thanks for the detailed response.

I had tried option 2 (using tcpdump to break down the files) and that indeed works. Option 3 (extra user-provided libpcap filter) is a better alternative though (and better also that my user-provided port idea, which was more limited), as it allows us to us the bigger files as-is. I had a go at implementing the extra filter and it works fine, all the way up to python. I'll open a PR with the changes.

Option 1 (using udpreplay) might be an alternative in the future, but the shortcomings you mention (full loading of the file, no endpoint specifications) mean we'd have to invest a bit more of time with that. There was also my implicit expectation that we could embed this into our sender program, but spawning a new process would be fine TBH.

Option 4 seems like a no-go given all the issues it has, but I'll keep it in mind for the future.