Performance: Eliminate obvious overhead from metrics computation

[cmalinmayor]

At a minimum, get back to speed that Ben experienced previously. @bentaculum can you put the commit hash here for reference?

[bentaculum]

Sure, here it is, from #57. f8c982c. I don't know how the current main compares to this, we'd have to run the benchmarks once retroactively on that commit somehow.

[cmalinmayor]

Thanks @bentaculum! I plan to manually run the benchmarking to debug the issues I have introduced and make a manual comparison, I'll add that here when I have it. Maybe adding a profiling tool at this point is worth the effort - I'll look into it as well. Of course, if you get to it first, the more the merrier :)