Add development chain-spec file for minimal/parachain templates for Omni Node compatibility

[EleisonC]

Description

This PR adds development chain specs for the minimal and parachain templates. #6334

Integration

This PR adds development chain specs for the minimal and para chain template runtimes, ensuring synchronization with runtime code. It updates zombienet-omni-node.toml, zombinet.toml files to include valid chain spec paths, simplifying configuration for zombienet in the parachain and minimal template.

Review Notes

1. Overview of Changes:

• Added development chain specs for use in the minimal and parachain template.
• Updated zombienet-omni-node.toml and zombinet.toml files in the minimal and parachain templates to include paths to the new dev chain specs.

2. Integration Guidance:
   NB: Follow the templates' READMEs from the polkadot-SDK master branch. Please build the binaries and runtimes based on the polkadot-SDK master branch.

• Ensure you have set up your runtimes parachain-template-runtime and minimal-template-runtime
• Ensure you have installed the nodes required ie parachain-template-node and minimal-template-node
• Set up Zombinet
• For running the parachains, you will need to install the polkadot cargo install --path polkadot remember from the polkadot-SDK master branch.
• Inside the template folders minimal or parachain, run the command to start with Zombienet with Omni Node, Zombienet with minimal-template-node or Zombienet with parachain-template-node

Include your leftover TODOs, if any, here.

• Test the syncing of chain specs with runtime's code.

[cla-bot-2021]

User @EleisonC, please sign the CLA here.

[iulianbarbu]

Thanks @EleisonC ! Left a few comments for now.
Looking forward to the README updates & btw, have you clarified how to make a rust test for the chain specs?

[EleisonC]

Thanks @EleisonC ! Left a few comments for now. Looking forward to the README updates & btw, have you clarified how to make a rust test for the chain specs?

Hey @iulianbarbu that is what I am looking into today. Do you have any pointers you'd like me to know as I move forward?

[iulianbarbu]

Thanks @EleisonC ! Left a few comments for now. Looking forward to the README updates & btw, have you clarified how to make a rust test for the chain specs?

Hey @iulianbarbu that is what I am looking into today. Do you have any pointers you'd like me to know as I move forward?

From the top of my head I thought about adding an integration test. You could build a ChainSpec as described here: https://github.com/paritytech/polkadot-sdk/blob/master/templates/parachain/node/src/chain_spec.rs#L36, and then call as_json -> and then convert it to serde_json::Value, and then read the newly added file and load its content as a serde_json::Value and then compare the two.

[EleisonC]

Hey @iulianbarbu one last question it's okay if I placed the test folder/file under the nodes folder/file. if not is there a better place

[iulianbarbu]

Hey @iulianbarbu one last question it's okay if I placed the test folder/file under the nodes folder/file. if not is there a better place

Hmm, thought about placing it under runtime initially. I think it makes more sense to place it under runtime so that if anything changes here, running the tests for the crate would fail because the changes will not be inline with dev_chain_spec.json. The node uses the logic defined in the runtime, so it would benefit from the runtime test, but not directly. I don't think we need to add a test on the same lines in the templates' nodes crate either because these nodes aren't depending on the dev_chain_spec.json file. Only OmniNode depends on this file being in sync with the runtime, and that not because it would fail otherwise, but because it would start an outdated version.

[EleisonC]

From the top of my head I thought about adding an integration test. You could build a ChainSpec as described here: https://github.com/paritytech/polkadot-sdk/blob/master/templates/parachain/node/src/chain_spec.rs#L36, and then call as_json -> and then convert it to serde_json::Value, and then read the newly added file and load its content as a serde_json::Value and then compare the two.

Hey @iulianbarbu regarding comparing the two I am a bit stuck on what to compare. From my research I would think it's other the code or value of top
Is it the value of code? Because the test fails when I compare the code value of the two chain spec files.
let test_runtime_code = test_json_value // .get("genesis") // .expect("failed to extract genesis") // .get("runtimeGenesis") // .expect("failed to extract runtimeGenesis from genesis") // .get("code") // .expect("failed to extract code from runtimeGenesis");
But the code value of test_runtime and the existing file do not match. Hence wondering if it would be the Top value to or maybe I am missing a step of conversion somewhere

[EleisonC]

From the top of my head I thought about adding an integration test. You could build a ChainSpec as described here: https://github.com/paritytech/polkadot-sdk/blob/master/templates/parachain/node/src/chain_spec.rs#L36, and then call as_json -> and then convert it to serde_json::Value, and then read the newly added file and load its content as a serde_json::Value and then compare the two.

Hey @iulianbarbu regarding comparing the two I am a bit stuck on what to compare. From my research I would think it's other the code or value of top Is it the value of code? Because the test fails when I compare the code value of the two chain spec files. let test_runtime_code = test_json_value // .get("genesis") // .expect("failed to extract genesis") // .get("runtimeGenesis") // .expect("failed to extract runtimeGenesis from genesis") // .get("code") // .expect("failed to extract code from runtimeGenesis"); But the code value of test_runtime and the existing file do not match. Hence wondering if it would be the Top value to or maybe I am missing a step of conversion somewhere

I figured out my problem was around decoding a Hex String to bytes. I can now properly traverse the values of the Top section and find the code for that spec file

[iulianbarbu]

From the top of my head I thought about adding an integration test. You could build a ChainSpec as described here: https://github.com/paritytech/polkadot-sdk/blob/master/templates/parachain/node/src/chain_spec.rs#L36, and then call as_json -> and then convert it to serde_json::Value, and then read the newly added file an

d load its content as a serde_json::Value and then compare the two.

Hey @iulianbarbu regarding comparing the two I am a bit stuck on what to compare. From my research I would think it's other the code or value of top Is it the value of code? Because the test fails when I compare the code value of the two chain spec files. let test_runtime_code = test_json_value // .get("genesis") // .expect("failed to extract genesis") // .get("runtimeGenesis") // .expect("failed to extract runtimeGenesis from genesis") // .get("code") // .expect("failed to extract code from runtimeGenesis"); But the code value of test_runtime and the existing file do not match. Hence wondering if it would be the Top value to or maybe I am missing a step of conversion somewhere

I figured out my problem was around decoding a Hex String to bytes. I can now properly traverse the values of the Top section and find the code for that spec file

Happy to hear you've sorted it out! I've noticed that in some tests we don't compare the code portions of the generated chain specs, and that's mainly because they are not super relevant for the test case. For this test though I think that it is relevant, to ensure generated chain specs correspond to the runtimes in the CI.

Some other good news is that once we'll have these chain specs we'll be able to easily run in the CI the omni_node_* tests like this one: https://github.com/paritytech/polkadot-sdk/blob/master/templates/zombienet/tests/smoke.rs#L160. We can update them in a follow up, no need to consider them right now, but it is up to you.

As usual, LMK in case of blockers, happy to take a look.

[EleisonC]

Hey @iulianbarbu,
I added a test and would appreciate it if you could look at it and approve it before I add the one for the para chain. Additionally, I added the test folder to nodes because I was encountering setup issues.

[iulianbarbu]

Can you please regenerate the dev_chain_spec.json for minimal (for now) after building the minimal/runtime with cargo build? I wonder if you built the runtime with --release and then generated the dev_chain_spec.json. It makes sense according to the README, but we run the tests with the testnet profile, which enables the debug symbols, and I think the chain spec code section differs based on the profile. I would be curious to run the test again after you regenerate the dev_chain_spec.json, and also see if regenerating on my machine results in a different code section, or chain spec altogheter.

[EleisonC]

Hey @iulianbarbu, I’d love to hear your thoughts on my approach and whether it aligns with the expectations.
cc: @skunert – I would appreciate your perspective as well.

[iulianbarbu]

Hi @EleisonC ! This is very nice and looks about done, except a few things that can be improved.

I plan to sync the templates with stable2412 release today & hopefully this will work without significant issues. Once that's done you can read every template readme as if it would be part of the dedicated template repos. This is just as FYI, I don't think you need to change more than what I suggested with this review, for this PR.

Other than this, would be great to open an issue related to the script we've discussed about, where we'll apply all this learnings about how to generate reproducible chain specs, if needed.

[iulianbarbu]

Other than this I plan to sync the templates with stable2412 release today & hopefully this will work without significant issues.

Can't be done yet. Sync job is still failing because it tries to update the templates' repos Cargo.toml deps versions based on the stable2412 tag, which doesn't have a commited Plan.toml (it was added after the tag was made). We'll need to wait for the stable2412-1 planned for next week, its tag, and then try to sync again.

[skunert]

Looking really good, only minor nits!

Thanks again for seeing this through :)!

[EleisonC]

Looking really good, only minor nits!

Thanks again for seeing this through :)!

Thank you for your guidance and feedback! I've addressed the minor nits. 😊

[skunert]

Looking good to me now! @iulianbarbu please verify the introduced workflow once more when you are back, then we are ready to go!

[skunert]

Here you are using matrix.package_name instead of matrix.template, I think this is a mismatch.

[EleisonC]

Thanks for the catch. I have addressed it

[iulianbarbu]

Just felt we can make the docs a bit clearer, so left a few clarifying comments, besides a few others for the misc-sync-templates.yml. It is mostly there :D

[iulianbarbu]

Exactly the same as for minimal

[EleisonC]

Hey @iulianbarbu done with the feedback.

[iulianbarbu]

Looks great! Thank you for sticking through this, it wasn't an easy one. :) I'll merge once you make the change for the README.docify.md.

[iulianbarbu]

nit:

Suggested change

	runtime_wasm_path: 'parachain-template-runtime/parachain_template_runtime.compact.compressed.wasm'
	runtime_wasm_path: parachain-template-runtime/parachain_template_runtime.compact.compressed.wasm

[iulianbarbu]

nit:

Suggested change

[iulianbarbu]

nit:

Suggested change

[iulianbarbu]

I recall we added this file for the parachain-template: https://github.com/paritytech/polkadot-sdk/blob/master/templates/parachain/README.docify.md, which should help with keeping in sync the README.md with code sections from parachain template components.

To make README.md changes for parachain template now we'll need to update the README.docify.md exactly as you updated the README.md, and then run (which will regenerate the README.md):

cd templates/parachain && cargo check --features generate-readme

LMK if you encounter issues with this.