Performance Comparison of Type 1 zkEVM with Other ZK Solutions

[lightning-li]

Hi Polygon Zero team, I'm a developer from the opBNB team. We're planning to develop a ZK fault proof for opBNB/Optimism based on the Type 1 zkEVM. To demonstrate the performance advantages of your Type 1 zkEVM, we developed a Proof of Concept (PoC) and compared its performance with RiscZero's zeth. Our performance test results show:

• ZK Cycles: Type 1 zkEVM is more efficient than RiscZero's zeth. For the same batch, zeth generates 10 times more ZK cycles than Type 1 zkEVM.
• Proving Time: Considering fixed server costs, Type 1 zkEVM (CPU version) is 5-6 times faster than zeth (GPU version).

More details are available in our blog post: https://forum.bnbchain.org/t/zk-fault-proof-for-opbnb/2915

Recently, the SP1 team released a blog comparing their performance with RiscZero: https://blog.succinct.xyz/sp1-benchmarks-8-6-24/. Their findings show:

• ZK Cycles: RiscZero's zeth generates 10 times more ZK cycles than SP1's zeth.
• Proving Time: SP1 (GPU) is 10 times faster than zeth (GPU).

Our tests used risc-zkvm v0.20.1 for zeth, while SP1 used v1.0.0. We understand v1.0.0 is 4 times faster than v0.20.1. Based on this information, we conclude:

• ZK Cycles: Type 1 zkEVM generates about the same number of ZK cycles as SP1's zeth.
• Proving Time: Considering fixed server costs, SP1's zeth (GPU version) is about 6 times faster than Type 1 zkEVM (CPU version).

We believe one reason Type 1 zkEVM is slower than SP1 is its use of Plonky2 on CPU, while SP1 uses Plonky3 on GPU. Here are some Questions:

1. What's your team's opinion on these performance comparisons with SP1?
2. Do you plan to migrate from Plonky2 to Plonky3 to leverage SP1's GPU work?
3. If Type 1 zkEVM used the same Plonky3 library, do you think it would outperform SP1's zeth?
4. Could you share your thoughts on the performance of Type 1 zkEVM (custom circuit) versus SP1 (generic VM approach)?

We appreciate any insights your team can provide on these matters.

[Nashtare]

Hi @lightning-li, thanks for reaching out and opening this discussion!

To get a better understanding of the benchmarks, may I know which version / commit revision of our Type 1 zkEVM that you used for your PoC?

Regarding your questions:

1. I'd expect, given the numbers that you have shared, that these have been taken without continuations, which we now support (at least an initial, sub-optimal version which will be released in the coming weeks. You can have a peek if you want at the feat/continuations branch. We expect to be 4x/5x cheaper than SP1 with this.

2. Regardless of SP1's GPU work, this may be a mid/long term plan indeed, especially as Plonky3 also supports Goldilocks for an initial smooth transition, while allowing us to benefit from all the nice upgrades that come with the toolkit. But no official decision nor timeline has been made.

3. The Type 1 already outperforms SP1 without even leveraging Plonky3 library, due to its very nature of being a non-generic zkVM, and also thanks to the brilliant minds working in the team :) I'd expect conservatively that switching to Plonky3, assuming it had a recursion approach similar to the one of plonky2, could yield a x2/x3 speed-up over what we currently have (which means x10/x15 over SP1), and even further if we would proceed to switching to a smaller field like BabyBear or Mersenne31, that could easily bring another x2/x3 improvement speed-up, cumulated to the previous one I mentioned.

4. Custom vs generic is a matter of practicality / cost efficiency / overall application. If your main concern is proving cost, then you'll probably always be better off dedicating additional resources towards a custom circuit approach, at least for the time being where there are still at least an order of magnitude between the two approaches. If your concern is quick and easy go-to-market / shipping, then the latter if preferred for example.

[lightning-li]

Hi @Nashtare, Thank you for your detailed reply. I appreciate the information you've provided. Allow me to address your points and share our perspective:

1. We used commit ID b7e1774 for Type1 zkEVM, which doesn't include the continuations feature and other performance improvements you team recently made. We agree that enabling this feature would likely improve Type1 zkEVM's performance. We plan to conduct another round of performance testing with this feature enabled. Regarding your expectation of Type1 zkEVM being 4x/5x cheaper than SP1, could you clarify if this comparison assumes both systems running on CPU/GPU, or Type1 zkEVM on CPU and SP1 on GPU? We believe a fair comparison would involve running both systems on their optimal hardware configurations.
2. We appreciate your openness about potential future changes. We look forward to receiving more information once your team makes an official decision.
3. From our findings, Type1 zkEVM generates the same number of zk cycles as SP1. If Type1 zkEVM also uses the Plonky3 library, would its performance advantage stem from generating fewer constraints per row with custom ASM instructions compared to RISC-V? Please correct me if my understanding is inaccurate.
4. Proving cost remains our primary concern, which initially led us to choose Type1 zkEVM. However, SP1's recent rapid performance improvements have prompted us to reassess our decision. Our primary concern is regarding the future performance of Type1 zkEVM compared to SP1. Specifically: If Type1 zkEVM doesn't migrate its proving system to Plonky3 in the near future, it may not be able to leverage the GPU optimizations that SP1 has implemented. So even with the addition of the continuations feature and other performance improvements to Type1 zkEVM, there's a risk that its performance (plonky2 running on CPU) might not surpass that of SP1 (plonky3 running on GPU). We will conduct further performance testing on Type1 zkEVM to make a final, informed decision.

We greatly appreciate your team's brilliant work. Once we have new performance data, we'll be sure to share it with you. We look forward to your continued impressive developments in this field.