Tracking issue: Performance benchmarks and profiling

[Davidson-Souza]

We would definitely benefit from some benchmarking and profiling of the current code, to find bottlenecks and choke points. This issue will keep track of them.

Here are some suggestions of things that could give us a good insight:

• Flame graphs: A simple, yet powerful profiling that can help us find places that are bottlenecking our code.
• Code benchmarks: The first step was made in Add criterion benches and restructure testdata #259, including more benchmarks would be very useful
• Memory profiling: Look for how we use memory and CPU cache. I would divide this into two categories:
  • heap allocation
  • cache friendless
• Figure out how well our current I/O code behaves, and if we can reduce how much we syscall/wait for data
• (anything else??)

[Davidson-Souza]

I've captured two flamegraphs on 6a286a6. Both are captured during block download, but one is while downloading mainly empty blocks, and the other after 350k, where it gets more full.

Both flamegraphs are with the default assumevalid, and I'm using --connect with another node running on my own home network.

It's funny how in the first one, our logger is a big resource taker. I guess we should go back to only logging every 10k blocks or something.