Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FRAME: Reintroduce TransactionExtension as a replacement for SignedExtension #3685

Open
wants to merge 133 commits into
base: master
Choose a base branch
from

Conversation

georgepisaltu
Copy link
Contributor

@georgepisaltu georgepisaltu commented Mar 13, 2024

Original PR #2280 reverted in #3665

This PR reintroduces the reverted functionality with additional changes, related effort here. Description is copied over from the original PR

First part of Extrinsic Horizon

Introduces a new trait TransactionExtension to replace SignedExtension. Introduce the idea of transactions which obey the runtime's extensions and have according Extension data (né Extra data) yet do not have hard-coded signatures.

Deprecate the terminology of "Unsigned" when used for transactions/extrinsics owing to there now being "proper" unsigned transactions which obey the extension framework and "old-style" unsigned which do not. Instead we have General for the former and Bare for the latter. (Ultimately, the latter will be phased out as a type of transaction, and Bare will only be used for Inherents.)

Types of extrinsic are now therefore:

  • Bare (no hardcoded signature, no Extra data; used to be known as "Unsigned")
    • Bare transactions (deprecated): Gossiped, validated with ValidateUnsigned (deprecated) and the _bare_compat bits of TransactionExtension (deprecated).
    • Inherents: Not gossiped, validated with ProvideInherent.
  • Extended (Extra data): Gossiped, validated via TransactionExtension.
    • Signed transactions (with a hardcoded signature).
    • General transactions (without a hardcoded signature).

TransactionExtension differs from SignedExtension because:

  • A signature on the underlying transaction may validly not be present.
  • It may alter the origin during validation.
  • pre_dispatch is renamed to prepare and need not contain the checks present in validate.
  • validate and prepare is passed an Origin rather than a AccountId.
  • validate may pass arbitrary information into prepare via a new user-specifiable type Val.
  • AdditionalSigned/additional_signed is renamed to Implicit/implicit. It is encoded for the entire transaction and passed in to each extension as a new argument to validate. This facilitates the ability of extensions to acts as underlying crypto.

There is a new DispatchTransaction trait which contains only default function impls and is impl'ed for any TransactionExtension impler. It provides several utility functions which reduce some of the tedium from using TransactionExtension (indeed, none of its regular functions should now need to be called directly).

Three transaction version discriminator ("versions") are now permissible (RFC here) in extrinsic version 5:

  • 0b00000101: Bare (used to be called "Unsigned"): contains Signature or Extra (extension data). After bare transactions are no longer supported, this will strictly identify an Inherents only.
  • 0b10000101: Old-school "Signed" Transaction: contains Signature, Extra (extension data) and an extension version byte, introduced as part of RFC99.
  • 0b01000101: New-school "General" Transaction: contains Extra (extension data) and an extension version byte, as per RFC99, but no Signature.

For the New-school General Transaction, it becomes trivial for authors to publish extensions to the mechanism for authorizing an Origin, e.g. through new kinds of key-signing schemes, ZK proofs, pallet state, mutations over pre-authenticated origins or any combination of the above.

UncheckedExtrinsic still maintains encode/decode backwards compatibility with extrinsic version 4, where the first byte was encoded as:

  • 0b00000100 - Unsigned transactions
  • 0b10000100 - Old-school Signed transactions, without the extension version byte

Code Migration

NOW: Getting it to build

Wrap your SignedExtensions in AsTransactionExtension. This should be accompanied by renaming your aggregate type in line with the new terminology. E.g. Before:

/// The SignedExtension to the basic transaction logic.
pub type SignedExtra = (
	/* snip */
	MySpecialSignedExtension,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>;

After:

/// The extension to the basic transaction logic.
pub type TxExtension = (
	/* snip */
	AsTransactionExtension<MySpecialSignedExtension>,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>;

You'll also need to alter any transaction building logic to add a .into() to make the conversion happen. E.g. Before:

fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let extra: SignedExtra = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */),
	);
	let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		extra,
	)
}

After:

fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let tx_ext: TxExtension = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */).into(),
	);
	let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		tx_ext,
	)
}

SOON: Migrating to TransactionExtension

Most SignedExtensions can be trivially converted to become a TransactionExtension. There are a few things to know.

  • Instead of a single trait like SignedExtension, you should now implement two traits individually: TransactionExtensionBase and TransactionExtension.
  • Weights are now a thing and must be provided via the new function fn weight.

TransactionExtensionBase

This trait takes care of anything which is not dependent on types specific to your runtime, most notably Call.

  • AdditionalSigned/additional_signed is renamed to Implicit/implicit.
  • Weight must be returned by implementing the weight function. If your extension is associated with a pallet, you'll probably want to do this via the pallet's existing benchmarking infrastructure.

TransactionExtension

Generally:

  • pre_dispatch is now prepare and you should not reexecute the validate functionality in there!
  • You don't get an account ID any more; you get an origin instead. If you need to presume an account ID, then you can use the trait function AsSystemOriginSigner::as_system_origin_signer.
  • You get an additional ticket, similar to Pre, called Val. This defines data which is passed from validate into prepare. This is important since you should not be duplicating logic from validate to prepare, you need a way of passing your working from the former into the latter. This is it.
  • This trait takes a Call type parameter. Call is the runtime call type which used to be an associated type; you can just move it to become a type parameter for your trait impl.
  • There's no AccountId associated type any more. Just remove it.

Regarding validate:

  • You get three new parameters in validate; all can be ignored when migrating from SignedExtension.
  • validate returns a tuple on success; the second item in the tuple is the new ticket type Self::Val which gets passed in to prepare. If you use any information extracted during validate (off-chain and on-chain, non-mutating) in prepare (on-chain, mutating) then you can pass it through with this. For the tuple's last item, just return the origin argument.

Regarding prepare:

  • This is renamed from pre_dispatch, but there is one change:
  • FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM validate!!
  • (This is different to SignedExtension which was required to run the same checks in pre_dispatch as in validate.)

Regarding post_dispatch:

  • Since there are no unsigned transactions handled by TransactionExtension, Pre is always defined, so the first parameter is Self::Pre rather than Option<Self::Pre>.

If you make use of SignedExtension::validate_unsigned or SignedExtension::pre_dispatch_unsigned, then:

  • Just use the regular versions of these functions instead.
  • Have your logic execute in the case that the origin is None.
  • Ensure your transaction creation logic creates a General Transaction rather than a Bare Transaction; this means having to include all TransactionExtensions' data.
  • ValidateUnsigned can still be used (for now) if you need to be able to construct transactions which contain none of the extension data, however these will be phased out in stage 2 of the Transactions Horizon, so you should consider moving to an extension-centric design.

@paritytech-review-bot paritytech-review-bot bot requested a review from a team March 13, 2024 19:39
@georgepisaltu georgepisaltu changed the title George/restore gav tx ext FRAME: Reintroduce TransactionExtension as a replacement for SignedExtension Mar 13, 2024
@georgepisaltu georgepisaltu added the T1-FRAME This PR/Issue is related to core FRAME, the framework. label Mar 13, 2024
@paritytech paritytech deleted a comment from command-bot bot Sep 13, 2024
@georgepisaltu
Copy link
Contributor Author

bot cancel 5-478ae05b-f12d-4078-a562-184d6fbfbb48

@command-bot
Copy link

command-bot bot commented Sep 13, 2024

@georgepisaltu Command "$PIPELINE_SCRIPTS_DIR/commands/bench-all/bench-all.sh" --target_dir=substrate has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/7344421 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/7344421/artifacts/download.

@georgepisaltu
Copy link
Contributor Author

bot bench-all substrate

@command-bot
Copy link

command-bot bot commented Sep 13, 2024

@georgepisaltu https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/7346730 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench-all/bench-all.sh" --target_dir=substrate. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 2-5d79bfe9-88ee-4d07-b1ec-c76f92205de0 to cancel this command or bot cancel to cancel all commands in this pull request.

@command-bot
Copy link

command-bot bot commented Sep 13, 2024

@georgepisaltu Command "$PIPELINE_SCRIPTS_DIR/commands/bench-all/bench-all.sh" --target_dir=substrate has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/7346730 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/7346730/artifacts/download.

Copy link
Member

@bkchr bkchr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still did not reviewed everything. However, wanting to give you this feedback already to give you time to fix them.

There are a lot of comments that you don't have addressed, from other peoples before. This needs to be done!

I found some extensions which are still written incorrectly, basically rejecting any kid of tx that is not using a system origin.

Cargo.lock Outdated Show resolved Hide resolved
substrate/frame/sudo/src/benchmarking.rs Outdated Show resolved Hide resolved
substrate/frame/transaction-payment/src/benchmarking.rs Outdated Show resolved Hide resolved
Comment on lines +1594 to +1610
/// Interface to differentiate between Runtime Origins authorized to include a transaction into the
/// block, and those who aren't.
///
/// Typically, upon validation or application of a transaction, the origin resulting from the
/// transaction extension (see [`TransactionExtension`]) is checked for authorization. Transaction
/// is then rejected or applied.
///
/// In FRAME, an authorized origin is either an `Origin::Signed` System origin or a custom origin
/// authorized in a `TransactionExtension`.
pub trait AsAuthorizedOrigin {
/// Whether the origin is authorized to include the transaction in a block.
///
/// In FRAME returns `false` if the origin is a System `Origin::None` variant, `true` otherwise,
/// meaning only signed or custom origin resulting from the transaction extension pipeline are
/// authorized.
fn is_authorized(&self) -> bool;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this? Maybe just using Option or a custom enum to represent the initial origin and then at the end just check if this was set?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes because at the substrate primitives level where TransactionExtension is defined, there is no concept of what an origin is, it's just a type. This trait is the way to differentiate between what the implementer decides is the "default" origin for their substrate runtime (if they're using a frame_system based runtime, it will probably be frame_system::RawOrigin::None BUT not necessarily). frame_system and frame_support are at a higher level in terms of dependency and we can't use concepts in those crates. As you can see in the VerifyMultiSignature implementation now, this trait is useful.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but as I said the initial origin could just be a None? This way, the origin can stay generic and we don't need this trait. Or what do I miss?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see one drawbacks in using Option<Origin>, transaction extensions can't add stuff on the origin prior to mutate it to an authorized origin.

Like an extension adding some filter.
Or if Origin type get more complex, some additional information on the origin.

That said such extension could always be put after authorizing transaction extensions in the pipeline.
So Option<Origin> is also ok to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Option<Origin> is redundant because all origin types used in practice, including frame_system as well as downstream custom ones, must have a value equivalent to "no origin" already. This would just introduce a "no origin for extension" (None) and then possibly a "no origin for call" (Some(frame_system::RawOrigin::None)).

In addition to what @gui1117 said, it's just more overhead for people writing extensions because it's explicit in the fn arguments and you have to "unwrap" it all the time and then get your relevant origin. With the trait, the impl is done once and you call it only if you care about this origin being authorized or not. Also, the trait allows users to have more origin values that are not authorized, or values that are authorized only in certain conditions, it's only limited by the trait impl.

Since the origin itself is a generic type, it makes sense we'd have a unifying interface for determining what's authorized and what's not. The trait is simple, the docs are clear, the impl is done for you and you bring it in context only if you use it, I don't see why not use this.

Copy link
Contributor

@gui1117 gui1117 Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we remove ValidateUnsigned we get: bare and general extrinsics start with frame system none origin. Bare extrinsics are only inherents, so they are inherently valid, and nodes include them into a block.
General extrinsics needs to be authorized to be valid and get into a block.

Seems coherent.

EDIT: I agree AsAuthorizedOrigin trait is not so much more complicated and probably more future proof.

gui1117 and others added 15 commits September 18, 2024 15:13
…extension PR (#5440)

* rename metadata stuff from `additional_signed` to `implicit` (doesn't
break metadata, only temporary representation in the code)
* removed some unused associated type `SignaturePayload`
* remove `From<u64>` from transaction payment transaction extension
* remove `TransactionExtensionBase`, merged into `TransactionExtension`

---------

Co-authored-by: georgepisaltu <[email protected]>
Signed-off-by: georgepisaltu <[email protected]>
Signed-off-by: georgepisaltu <[email protected]>
Signed-off-by: georgepisaltu <[email protected]>
Signed-off-by: georgepisaltu <[email protected]>
Signed-off-by: georgepisaltu <[email protected]>
Signed-off-by: georgepisaltu <[email protected]>
Signed-off-by: georgepisaltu <[email protected]>
Signed-off-by: georgepisaltu <[email protected]>
@georgepisaltu
Copy link
Contributor Author

There are a lot of comments that you don't have addressed, from other peoples before.

I think everything from other reviewers (so far) has been addressed now.

I found some extensions which are still written incorrectly, basically rejecting any kid of tx that is not using a system origin.

As outlined in other comments on the issue, I thought that we would merge this PR without support for "other" custom origins and we would add support for that later. We are not currently using this in our codebase and we will only when we merge phase 2 of Extrinsic Horizon, but regardless I will work on allowing custom origins in all these extensions because it shouldn't be a problem now that we reject transactions without an origin by default.

@bkchr
Copy link
Member

bkchr commented Sep 18, 2024

but regardless I will work on allowing custom origins in all these extensions because it shouldn't be a problem now that we reject transactions without an origin by default.

IMO, if we touch the code now any way and rewrite them as transaction extension, we should make them work properly. Otherwise we will miss them later.

cumulus/parachains/runtimes/starters/shell/src/lib.rs Outdated Show resolved Hide resolved
polkadot/runtime/common/src/claims.rs Outdated Show resolved Hide resolved
substrate/test-utils/runtime/src/lib.rs Show resolved Hide resolved
substrate/test-utils/runtime/src/extrinsic.rs Show resolved Hide resolved
substrate/primitives/runtime/src/testing.rs Show resolved Hide resolved
pub fn new_unsigned(function: Call) -> Self {
Self { signature: None, function }
/// Returns `true` if this extrinsic instance is an inherent, `false`` otherwise.
pub fn is_inherent(&self) -> bool {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record this naming is not meaningful until we remove ValidateUnsigned.

We could name it is_bare during the transition period. But it is also ok to use the final naming.

Comment on lines +598 to +606
/// Create new `SignedPayload` from raw components.
pub fn from_raw(call: Call, tx_ext: Extension, implicit: Extension::Implicit) -> Self {
Self((call, tx_ext, implicit), EXTRINSIC_FORMAT_VERSION)
}

/// Deconstruct the payload into it's components.
pub fn deconstruct(self) -> (Call, Extension, Extension::Implicit) {
self.0
}
Copy link
Contributor

@gui1117 gui1117 Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way this type is used is not so ideal to me.

Like here, executing deconstruct and then from_raw can change the actual value.

And in Encode we also lose the extrinsic format version.

Maybe having 2 types: SignedPayloadV4 and SignedPayloadV5 would make the code safer.

or new_v4, new_v5 etc..

But maybe it is fine as it is.

Copy link
Contributor

@gui1117 gui1117 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code is good to me.

But before merging we should find consensus on those discussions:

  • make all transaction extensions allow not signed origin to go through in this PR.
    IMO a follow-up is good. We can have a tracking issue, and it shouldn't be difficult.

  • AsAuthorizedOrigin trait or just using Option<Origin> in transaction extension pipeline.
    IMO AsAuthorizedOrigin trait is good, frame system None origin fits our usecase, and it allows us to do more stuff with origin in the pipeline. But I don't mind Option<Origin> either.

  • in frame system offchain rename CreateInherent to CreateBare or something, and also is_inherent to is_bare?
    IMO: CreateBare but overall I don't mind.

And some minor ones:

  • merge PR docs into one for this PR.

  • Extrinsic failed and extrinsic success events: use a type to avoid too much breaking change for people using events.

Comment on lines +120 to +121
- name: frame
bump: major
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: frame
bump: major
- name: polkadot-sdk-frame
bump: major

For CI

Comment on lines +98 to +103
- name: minimal-runtime
bump: major
- name: node-template
bump: major
- name: node-template-runtime
bump: major
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: minimal-runtime
bump: major
- name: node-template
bump: major
- name: node-template-runtime
bump: major
- name: minimal-template-runtime
bump: major
- name: minimal-template-node
bump: major
- name: solochain-template-runtime
bump: major

CI say those crates don't exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T1-FRAME This PR/Issue is related to core FRAME, the framework.
Projects
Status: Audited
Development

Successfully merging this pull request may close these issues.

7 participants