Batch Amendment #5060

dangell7 · 2024-07-10T15:45:43Z

High Level Overview of Change

Context of Change

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Refactor (non-breaking change that only restructures code)
Performance (increase or change in throughput and/or latency)
Tests (you added tests for code that already exists, or your new feature included in this PR)
Documentation update
Chore (no impact to binary, e.g. .gitignore, formatting, dropping support for older tooling)
Release

API Impact

Public API: New feature (new methods and/or new fields)
Public API: Breaking change (in general, breaking changes should only impact the next api_version)
libxrpl change (any change that may affect libxrpl or dependents of libxrpl)
Peer protocol change (must be backward compatible or bump the peer protocol version)

codecov · 2024-07-10T16:10:41Z

Codecov Report

Attention: Patch coverage is 90.02433% with 41 lines in your changes missing coverage. Please review.

Project coverage is 77.9%. Comparing base (7b18006) to head (04519e6).

Files with missing lines	Patch %	Lines
src/xrpld/app/tx/detail/Batch.cpp	94.2%	9 Missing ⚠️
src/xrpld/app/tx/detail/Transactor.cpp	87.1%	9 Missing ⚠️
src/xrpld/overlay/detail/PeerImp.cpp	0.0%	8 Missing ⚠️
src/libxrpl/protocol/STTx.cpp	90.9%	6 Missing ⚠️
src/xrpld/app/misc/NetworkOPs.cpp	66.7%	6 Missing ⚠️
src/libxrpl/protocol/TxMeta.cpp	83.3%	1 Missing ⚠️
src/xrpld/app/ledger/detail/OpenLedger.cpp	50.0%	1 Missing ⚠️
src/xrpld/app/tx/detail/apply.cpp	88.9%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #5060     +/-   ##
=========================================
+ Coverage     77.9%   77.9%   +0.1%     
=========================================
  Files          782     785      +3     
  Lines        66614   66964    +350     
  Branches      8159    8182     +23     
=========================================
+ Hits         51861   52174    +313     
- Misses       14753   14790     +37

Files with missing lines	Coverage Δ
include/xrpl/protocol/Batch.h	`100.0% <100.0%> (ø)`
include/xrpl/protocol/Feature.h	`100.0% <ø> (ø)`
include/xrpl/protocol/HashPrefix.h	`100.0% <ø> (ø)`
include/xrpl/protocol/STTx.h	`100.0% <ø> (ø)`
include/xrpl/protocol/TER.h	`100.0% <ø> (ø)`
include/xrpl/protocol/TxMeta.h	`93.5% <100.0%> (+1.9%)`	⬆️
include/xrpl/protocol/detail/transactions.macro	`100.0% <100.0%> (ø)`
src/libxrpl/protocol/InnerObjectFormats.cpp	`100.0% <100.0%> (ø)`
src/libxrpl/protocol/TER.cpp	`100.0% <ø> (ø)`
src/xrpld/app/tx/detail/ApplyContext.cpp	`100.0% <100.0%> (ø)`
... and 17 more

... and 5 files with indirect coverage changes

src/libxrpl/protocol/STTx.cpp

This is not finished. Needs to only revert the transactions submitted not the entire open ledger.

src/xrpld/app/tx/detail/Batch.cpp

ximinez · 2024-07-31T16:36:25Z

Also, since it seems this PR is still under active development, could you convert it to a draft?

- allow only one flag - validate batch signers array (max and valid) - move `calculateBaseFee` - move `preflight2` - no duplicate TxIDs - no duplicate BatchSigners

shawnxie999

partial

src/xrpld/app/tx/detail/apply.cpp

src/xrpld/app/tx/detail/Batch.cpp

shawnxie999

partial

src/xrpld/app/tx/detail/Batch.cpp

shawnxie999 · 2024-11-04T15:53:56Z

src/xrpld/app/tx/detail/Batch.cpp

+
+            if (flags & tfUntilFailure)
+            {
+                result = tesSUCCESS;


so it's possible that none of the inner txns succeeds and yet the batch txn is "successful"?

src/xrpld/app/tx/detail/Batch.cpp

src/xrpld/ledger/ApplyViewImpl.h

src/xrpld/app/tx/detail/ApplyContext.cpp

src/xrpld/ledger/ApplyViewImpl.h

src/xrpld/app/tx/detail/Batch.cpp

shawnxie999

partial

src/xrpld/app/tx/detail/Transactor.cpp

src/xrpld/app/tx/detail/ApplyContext.cpp

shawnxie999 · 2024-11-04T18:54:22Z

src/xrpld/app/tx/detail/ApplyContext.cpp

+void
+ApplyContext::applyOpenView(OpenView& open)
+{
+    if (!base_.open())


could you elaborate more on why we are applying when the base is closed, and not when open? perhaps it's helpful to put more into the comment as well. thanks!

src/xrpld/app/tx/detail/Batch.cpp

src/xrpld/app/misc/NetworkOPs.cpp

shawnxie999

have some questions regarding how inner txns states could've been applied

shawnxie999 · 2024-11-05T16:22:13Z

src/xrpld/app/tx/detail/Transactor.cpp

+        if (pkSigner.empty())
+        {
+            STArray const& txSigners(signer.getFieldArray(sfSigners));
+            ret = checkMultiSign(ctx.view, idAccount, txSigners, ctx.j);


Suggested change

ret = checkMultiSign(ctx.view, idAccount, txSigners, ctx.j);

if (ret = checkMultiSign(ctx.view, idAccount, txSigners, ctx.j);

!isTesSuccess(ret))

return ret;

currently, it would return code of the result of the last signature, but shouldn't we be returning the first error that occurs in the loop? If so, we should probably add a test case to exercise the behavior where the last signer signature is correct, but previous ones are bad.

Thanks for the suggestion. I will add a test case as well

shawnxie999 · 2024-11-05T16:22:23Z

src/xrpld/app/tx/detail/Transactor.cpp

+            if (!sleAccount)
+                return tesSUCCESS;
+
+            ret = checkSingleSign(


same as above

shawnxie999 · 2024-11-05T16:31:10Z

src/xrpld/app/tx/detail/Transactor.cpp

+    ApplyViewImpl& avi = dynamic_cast<ApplyViewImpl&>(ctx_.view());
+    std::vector<STObject> executions;
+    avi.copyBatchMetaData(executions);
    ctx_.discard();
+    ApplyViewImpl& avi2 = dynamic_cast<ApplyViewImpl&>(ctx_.view());
+    avi2.setBatchMetaData(std::move(executions));


what is this block of code here for?

This is to apply the batch executions to the metadata even in the event of a tec failure

we should add a comment above this code block as what you described.

also should we consider adding a if-condition where this block is executed conditionally only if the transaction type is Batch?

src/xrpld/app/tx/detail/Transactor.cpp

shawnxie999 · 2024-11-05T16:49:08Z

src/xrpld/app/tx/detail/Batch.cpp

+        if (not3rdParty)
+            ctx_.batchPrevious(avi);
+
+        ctx_.applyOpenView(subView);


Also just curious, does using subView.apply(ctx_.rawView()) not solve this problem? assuming that subView contains all the changes for the inner txns, shouldn't it include the latest account root state of the "non3rdParty" account? Please correct me if I'm wrong.

shawnxie999 · 2024-11-05T16:53:55Z

src/xrpld/app/tx/detail/Transactor.cpp

+        // Only update the account root entry if the batch transaction was
+        // not a 3rd party transaction
+        if (not3rdParty)
+            ctx_.updateAccountRootEntry();


linking to this comment thread

I'm still unclear why exactly we need to explicitly update account root entry here. Does it not work if we apply the state changes from subView inside the Batch transactor?

I get an error when I attempt to apply the subView to the open view. Maybe I'm doing something wrong?

- Update Comments - Update/Change function names - Update/Change argument names - Remove unused preclaim - Fix exit out of signature check (multi & single)

shawnxie999

partial

shawnxie999 · 2024-11-06T21:41:32Z

src/xrpld/ledger/detail/ApplyStateTable.cpp

@@ -147,7 +159,8 @@ ApplyStateTable::apply(
            }
            auto const origNode = to.read(keylet::unchecked(item.first));
            auto curNode = item.second.second;
-            if ((type == &sfModifiedNode) && (*curNode == *origNode))
+            if ((type == &sfModifiedNode) && (*curNode == *origNode) &&
+                !isBatch)


if i'm understanding correctly, when the transaction type is Batch, we are going to add the previous and final fields of the object even though they are the same.

This doesn't sound too correct, please correct me if i'm wrong. TY

We are going to add the batchPrevAcctRootFields. The curNode and origNode are the same on the batch txn because of the way its reapplied in updateAccountRootEntry. I guess handling this as its own case might be preferred for better understanding?

we can just add a comment here to explain the situation. Also it might be worth add an assertion to make sure that nodes are not modified if it is a batch transaction - this might before useful for future debugging

// If it is a batch txn, the curNode and origNode are the same on the batch txn // because of the way its reapplied in updateAccountRootEntry bool const nodesUnmodified = *curNode == *origNode; if ((type == &sfModifiedNode) && nodesUnmodified && !isBatch) continue; assert(isBatch && !nodesUnmodified)

shawnxie999 · 2024-11-06T21:50:47Z

src/xrpld/ledger/detail/ApplyStateTable.cpp

@@ -147,7 +159,8 @@ ApplyStateTable::apply(
            }
            auto const origNode = to.read(keylet::unchecked(item.first));
            auto curNode = item.second.second;
-            if ((type == &sfModifiedNode) && (*curNode == *origNode))
+            if ((type == &sfModifiedNode) && (*curNode == *origNode) &&
+                !isBatch)
                continue;
            std::uint16_t nodeType = curNode


can a batch transaction ever modify an object other than AccountRoot? if not, would be use good to assert that here.

No I dont believe so. I will add that here.

shawnxie999 · 2024-11-06T21:52:33Z

src/xrpld/ledger/detail/ApplyStateTable.cpp

@@ -203,6 +216,13 @@ ApplyStateTable::apply(
                        prevs.emplace_back(obj);
                }

+                if (isBatch && nodeType == ltACCOUNT_ROOT && batchPrev)
+                {
+                    // TODO: This could fail if the fields already exist


is this TODO done? also would love to learn more about this, as the comment itself isn't too comprehensive 😃

- Remove BatchTxn Object - Use Global Flag `tfInnerBatchTxn ` for inner batch txns - Remove Sequence & Ticket workaround - replace sfBatchTxn submit/relay validation with tfInnerBatchTxn - Remove Sequence 0 Requirement

shawnxie999

partial

shawnxie999 · 2024-11-07T20:22:29Z

src/libxrpl/protocol/STTx.cpp

+            Serializer dataStart = startMultiSigningData(*this);
+            finishMultiSigningData(accountID, dataStart);
+            return dataStart.getData();
+        });


what's the rationale for not returning a Slice directly in this lambda?

shawnxie999 · 2024-11-07T20:27:01Z

include/xrpl/protocol/TxFlags.h

+constexpr std::uint32_t tfUniversalV2                      = tfFullyCanonicalSig | tfInnerBatchTxn;
+constexpr std::uint32_t tfUniversalMask                    = ~(tfFullyCanonicalSig | tfInnerBatchTxn);


Suggested change

constexpr std::uint32_t tfUniversalV2 = tfFullyCanonicalSig | tfInnerBatchTxn;

constexpr std::uint32_t tfUniversalMask = ~(tfFullyCanonicalSig | tfInnerBatchTxn);

constexpr std::uint32_t tfUniversal = tfFullyCanonicalSig | tfInnerBatchTxn;

constexpr std::uint32_t tfUniversalMask = ~(tfFullyCanonicalSig | tfInnerBatchTxn);

we can prob add tfInnerBatchTxn into tfUniversal directly, and check in Transactor::preflight1 if tfInnerBatchTxn is usable

include/xrpl/protocol/TxFlags.h

src/xrpld/ledger/ApplyViewImpl.h

src/xrpld/ledger/detail/ApplyStateTable.cpp

src/xrpld/ledger/detail/ApplyViewImpl.cpp

src/xrpld/ledger/ApplyViewImpl.h

src/xrpld/app/tx/detail/apply.cpp

shawnxie999

partial

shawnxie999 · 2024-11-11T15:00:42Z

src/xrpld/app/tx/detail/Transactor.cpp

+            return tesSUCCESS;
+        }
+        JLOG(ctx.j.warn()) << "Batch: sfFee must be zero.";
+        return temBAD_FEE;


we should add a test that checks if inner transaction has a specified non zero transaction fee

src/test/app/Batch_test.cpp

shawnxie999 · 2024-11-11T15:10:52Z

src/test/app/Batch_test.cpp

+        {
+            auto const seq = env.seq(alice);
+            auto const batchFee = feeDrops * 2;
+            env(batch::batch(alice, seq, batchFee, tfAllOrNothing),


should also add a test where batch transaction sets tfInnerBatchTxn

shawnxie999 · 2024-11-11T15:32:23Z

src/test/app/Batch_test.cpp

+                batch::add(pay(alice, bob, XRP(10)), seq + 0),
+                batch::add(pay(bob, alice, XRP(5)), env.seq(bob)),
+                batch::sig(carol),
+                ter(temBAD_SIGNER));


do we have a test where it alice's sig is also included in the BatchSigners? something like

env(batch::batch(alice, seq, batchFee, tfAllOrNothing), batch::add(pay(alice, bob, XRP(10)), seq + 0), batch::add(pay(bob, alice, XRP(5)), env.seq(bob)), batch::sig(alice, bob), ter(temBAD_SIGNER));

This should fail because alice's sig is already provided in the outer tx.

src/test/app/Batch_test.cpp

shawnxie999 · 2024-11-11T19:40:36Z

src/test/app/Batch_test.cpp

+        env(batch::batch(alice, seq, batchFee, tfAllOrNothing),
+            batch::add(pay(alice, bob, XRP(10)), seq + 1),
+            batch::add(pay(bob, alice, XRP(5)), env.seq(bob)),
+            batch::sig(bob));


could also possibly add a test where bob's signature is incorrect

Is this not the same test? https://github.com/XRPLF/rippled/pull/5060/files#diff-48ec7d1a37f3978d1c2e296a759734453fbc234e49cdbb1157e6da49a93fc325R314

i can't see which test the links points me to, i think it's a github bug. But what i mean when i say bad sig is when the TxnSignature has been tampered. do you think it's worth testing?

shawnxie999 · 2024-11-11T19:44:16Z

src/test/app/Batch_test.cpp

+        env(batch::batch(alice, seq, batchFee, tfAllOrNothing),
+            batch::add(pay(alice, bob, XRP(1)), seq + 1),
+            batch::add(pay(alice, bob, XRP(1)), seq + 2),
+            msig(bob, carol));


could have a test where either carol or bob's sig is bad

This is just a multisigned txn. But I will add those two edge cases.

shawnxie999 · 2024-11-11T19:44:37Z

src/test/app/Batch_test.cpp

+        env(batch::batch(alice, seq, batchFee, tfAllOrNothing),
+            batch::add(pay(alice, bob, XRP(10)), seq + 1),
+            batch::add(pay(bob, alice, XRP(5)), env.seq(bob)),
+            batch::msig(bob, {dave, carol}));


could also add a test where multi sign does not meet the quorum

also might be worth having one or two more BatchSigner, this test only has one BatchSigner

shawnxie999 · 2024-11-11T20:06:28Z

src/test/app/Batch_test.cpp

+    void
+    testNoAccount(FeatureBitset features)
+    {
+        testcase("no account");


is this the right name for the test? seems like the test is just submitting a batch with payment and accountset. there is a purpose for testing this combo specifically?

This test is submitting a payment to an account that does not exist and then doing the AccountSet. Its just there to test that account activation works. I can change the name? What name would you suggest? Or should I remove the test all together?

I see that makes sense, might be worth aadding a comment like

// Create Bob's account by funding it with XRP as an inner transaction. // Then Bob submits an AccountSet transaction as another inner transaction.

shawnxie999

partial

shawnxie999 · 2024-11-12T16:27:56Z

src/test/app/Batch_test.cpp

+            batch::add(pay(alice, bob, XRP(10)), preAliceSeq + 1),
+            batch::add(pay(bob, alice, XRP(5)), preBobSeq + 10),
+            batch::sig(bob),
+            ter(tecBATCH_FAILURE));


should add another test to check where alice specifies the same sequence for both outer and inner txns:

env(batch::batch(alice, preAliceSeq, batchFee, tfAllOrNothing), batch::add(pay(alice, bob, XRP(10)), preAliceSeq), batch::add(pay(bob, alice, XRP(5)), preBobSeq), batch::sig(bob), ter(tecBATCH_FAILURE));

shawnxie999 · 2024-11-12T17:29:53Z

src/xrpld/app/tx/detail/Transactor.cpp

+    ApplyViewImpl& avi = dynamic_cast<ApplyViewImpl&>(ctx_.view());
+    std::vector<STObject> executions;
+    avi.copyBatchMetaData(executions);
    ctx_.discard();
+    ApplyViewImpl& avi2 = dynamic_cast<ApplyViewImpl&>(ctx_.view());
+    avi2.setBatchMetaData(std::move(executions));


we should add a comment above this code block as what you described.

also should we consider adding a if-condition where this block is executed conditionally only if the transaction type is Batch?

dangell7 added 2 commits July 10, 2024 17:44

featureBatch rough draft

879aba5

clang-format

46fd68f

dangell7 added 13 commits July 17, 2024 20:33

add atomic swap

c57ace7

Merge branch 'develop' into feature-batch

91d9909

add batch bsig

f37377d

[fold] clang-format

b3136a2

[fold] remove tapRETRY

5e919eb

[fold] remove comments

1225a61

[fold] fix invariant workaround

304ff3e

[fold] fix applyTransaction workaround

aded948

[fold] remove test template

82d4943

[fold] update logging

eaf7893

[fold] remove log

ca39807

clang-format

dd71073

[fold] remove workaround

26518a5

cindyyan317 reviewed Jul 30, 2024

View reviewed changes

src/libxrpl/protocol/STTx.cpp Outdated Show resolved Hide resolved

dangell7 added 6 commits July 30, 2024 14:05

update atomic

3ceb05c

refactor & OpenView revert()

12324f1

This is not finished. Needs to only revert the transactions submitted not the entire open ledger.

[fold] clang-format

e9f045b

Merge branch 'develop' into feature-batch

51ed234

[fold] add revert

676b9b6

[fold] clang-format

6c6180d

ximinez reviewed Jul 31, 2024

View reviewed changes

src/xrpld/app/tx/detail/Batch.cpp Outdated Show resolved Hide resolved

dangell7 marked this pull request as draft July 31, 2024 19:53

dangell7 added 4 commits August 1, 2024 10:37

[revert] remove rawRevert func

bd03ea3

refactor with stacking views

c0e00b0

clang-format

a3e15a1

Merge branch 'develop' into feature-batch

14eef60

dangell7 added 13 commits October 30, 2024 22:09

[fold] update headers

767a88e

[fold] change error response text

ee03811

[fold] change flags bit

9d8368d

[fold] fix formatting

b408b80

[fold] normal consequences

27d4685

[fold] refactor single/multi sign

419b14d

[fold] review comments

6c87e70

- allow only one flag - validate batch signers array (max and valid) - move `calculateBaseFee` - move `preflight2` - no duplicate TxIDs - no duplicate BatchSigners

[fold] clang-format

a75cf0e

[fold] tickets are not chronological

e6021bf

[fold] fix fee calculation

fd0acb7

Merge branch 'develop' into feature-batch

8a6d6ce

[fold] fix merge issues

dd77e03

[fold] clang-format

6cb23e3

shawnxie999 reviewed Nov 2, 2024

View reviewed changes

src/xrpld/app/tx/detail/apply.cpp Outdated Show resolved Hide resolved

src/xrpld/app/tx/detail/Batch.cpp Outdated Show resolved Hide resolved

shawnxie999 reviewed Nov 4, 2024

View reviewed changes

shawnxie999 reviewed Nov 5, 2024

View reviewed changes

[fold] addressing review

88c40e4

- Update Comments - Update/Change function names - Update/Change argument names - Remove unused preclaim - Fix exit out of signature check (multi & single)

shawnxie999 reviewed Nov 6, 2024

View reviewed changes

dangell7 added 7 commits November 7, 2024 01:52

[fold] change sequence handling

e4cb784

- Remove BatchTxn Object - Use Global Flag `tfInnerBatchTxn ` for inner batch txns - Remove Sequence & Ticket workaround - replace sfBatchTxn submit/relay validation with tfInnerBatchTxn - Remove Sequence 0 Requirement

[fixup] rename prevFields -> batchPrevAcctRootFields

47120ac

[fold] revert applyFlags change

6ed4ac8

[fold] use unordered_set

52f1ba9

Merge branch 'develop' into feature-batch

1978fd3

[fold] clang-format

6ffe349

[fold] fix bad merge

04519e6

shawnxie999 reviewed Nov 7, 2024

View reviewed changes

mvadari reviewed Nov 7, 2024

View reviewed changes

src/xrpld/app/tx/detail/apply.cpp Show resolved Hide resolved

shawnxie999 reviewed Nov 11, 2024

View reviewed changes

shawnxie999 reviewed Nov 12, 2024

View reviewed changes

-            ret = checkMultiSign(ctx.view, idAccount, txSigners, ctx.j);
+            if (ret = checkMultiSign(ctx.view, idAccount, txSigners, ctx.j);
+                !isTesSuccess(ret))
+                return ret;

		constexpr std::uint32_t tfUniversalV2 = tfFullyCanonicalSig \| tfInnerBatchTxn;
		constexpr std::uint32_t tfUniversalMask = ~(tfFullyCanonicalSig \| tfInnerBatchTxn);

Batch Amendment #5060

Are you sure you want to change the base?

Batch Amendment #5060

Conversation

dangell7 commented Jul 10, 2024 • edited Loading

High Level Overview of Change

Context of Change

Type of Change

API Impact

codecov bot commented Jul 10, 2024 • edited Loading

Codecov Report

ximinez commented Jul 31, 2024

shawnxie999 left a comment

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

shawnxie999 Nov 4, 2024 • edited Loading

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dangell7 commented Jul 10, 2024 •

edited

Loading

codecov bot commented Jul 10, 2024 •

edited

Loading

shawnxie999 Nov 4, 2024 •

edited

Loading

shawnxie999 Nov 7, 2024 •

edited

Loading

shawnxie999 Nov 12, 2024 •

edited

Loading