What to do about benchmarks?

[hildjj]

In the big monorepo refactor, I got bogged down in making the benchmarks work. That code is not terribly modern, including using (new Date()).getTime() as it's source of information.

benchmark.js is under-documented and hasn't been updated in 4 years, so it's not my first choice. Does anyone have a benchmarking framework they like at the moment?

[StoneCypher]

For local stuff that isn't critical I use benny

For network stuff I use artillery for light stuff and tsung or jmeter for heavy stuff, but that isn't relevant to this question

[bd82]

benny looks pretty good. I think the goal here is to compare a proposed change to the existing codebase, so we'll need to do some post-processing of the json output. I don't want to save the results long-term, because I don't want to have to maintain a piece of reference hardware forever, but perhaps we can do something to automate doing git checkouts and generating the results for previous versions all on one machine.

My low-tech solution for this scenario is to run benchmarks in the browser and rely on unpkg.com to load the latest bundled released version
to compare against the "local dev branch".

• https://github.com/Chevrotain/chevrotain/tree/master/packages/chevrotain/benchmark_web

While it is low tech and un-automated it certainly helped avoid performance regressions and implement optimizations...

btw, I added Peggy to my JS parsing libraries benchmark and there seems to be a ~5% performance regression (on latest chorme)
versus PegJS.
I suppose this makes sense if some new features have been added...

• https://chevrotain.io/performance/

[Mingun]

btw, I added Peggy to my JS parsing libraries benchmark and there seems to be a ~5% performance regression (on latest chorme) versus PegJS.

This definitely very strange because if you compare both parsers the peggy one is smaller (you can do this in a browser using github1s.com or github.dev).

[bd82]

This definitely very strange because if you compare both parsers the peggy one is smaller

Why would the file size be directly correlated with performance?
It may be, but it does not have to be...

BTW when generating the peggy parser with version 1.0.0 of peggy the performance is virtually the same.
So perhaps the regression was introduced somewhere between 1.0.0 and 1.2.0

There may also be issues with the benchmark.
For example when testing different versions (master vs latest) of the Chevrotain library the isolation in iframes
was insufficient which forced me run in different tabs/browser windows and to share benchmark results (to compare) via the browser's local storage.

[Mingun]

I'm talking not about file size. If you look at diff, you find only two significant changes in the code:

• peggy removes many useless checks
• SyntaxError definition slightly changed. Because parsing completes not with an exception that means, that this changes is irrelevant because this code runs only when parsing fail

That is why I wonder, why pegjs shows the better results than peggy? Probably this is some JS engine related nuance, such as number of local functions (4 more in peggy)

[bd82]

I'm talking not about file size. If you look at diff, you find only two significant changes in the code:

Interesting, the diff does not indeed easily explain the performance difference.

You can see the diff between peggy 1.2.0 and 1.0.0 here:

• https://github.com/Chevrotain/chevrotain/compare/temp_diff?expand=1#diff-d7a0538ff316d5834723d688aba10698ab50da048a27fbd6eefba054ec4485eb

Perhaps it is indeed some internal V8 strange behavior, e.g the parser object now being recognized as a proper class.
For example of another strange performance behavior: Chevrotain dynamically modifies the Parser instance's properties/methods, and so some voodoo magic has to be done to help V8 understand it is holding a "real" constructor and that the fast property access should be done.

• https://github.com/Chevrotain/chevrotain/blob/239c59fd48ea650b21fb31b3f2492fe1aa865b44/packages/utils/src/api.ts#L443-L462

Or maybe the code is so similar that there is some internal V8 shared state and hot spot optimizations that
interact badly with each other (even though different IFrames are used).