why op-node always onto old blocks after recharge batcher

[Eoous]

Forget to charge batcher, after charging batcher restart batcher, op-node always build old headers

t=2024-09-21T20:32:53+0000 lvl=info msg="sequencer started building new block" payload_id=0x747b3cffdb20b41e l2_parent_block=0x1d8f3f6139ea6238e75bf79444a98f68ee5ddab78015058b33f37d3c71ec6638:2381876 l2_parent_block_time=1726944391
t=2024-09-21T20:32:53+0000 lvl=eror msg="L1 Info transaction differs" number=2345041 time=1726907556 safe_l1_number=44062297 safe_l1_hash=0x17ab1602193cd09273e9149282142da39d923e59c5584d1c4d801c1d4be3d9f7 safe_l1_time=1726907556 safe_seq_num=0 safe_l1_basefee=10000000000 safe_batcher_addr=0xd86EeFc60752ff29549F19b54DA42523b197B276 unsafe_l1_number=44062296 unsafe_l1_hash=0x8ccccd13f38a1e142efc851bf8ef48aa6702cc502b8215936d210bda465d7ffd unsafe_l1_time=1726907553 unsafe_seq_num=3 unsafe_l1_basefee=10725000000 unsafe_batcher_addr=0xd86EeFc60752ff29549F19b54DA42523b197B276 safe_l1_blob_basefee=1 safe_l1_basefee_scalar=1000000 safe_l1_blob_basefee_scalar=0 unsafe_l1_blob_basefee=1 unsafe_l1_basefee_scalar=1000000 unsafe_l1_blob_basefee_scalar=0
t=2024-09-21T20:32:53+0000 lvl=warn msg="L2 reorg: existing unsafe block does not match derived attributes from L1" err="transaction 0 does not match. expected: 0x7ef8f8a09b31622e432b930f1763b7d760e725793b2978af0325766c0867e1343176ad5a94deaddeaddeaddeaddeaddeaddeaddeaddead00019442000000000000000000000000000000000000158080830f424080b8a4440a5e20000f42400000000000000000000000030000000066ee84a10000000002a05658000000000000000000000000000000000000000000000000000000027f42834000000000000000000000000000000000000000000000000000000000000000018ccccd13f38a1e142efc851bf8ef48aa6702cc502b8215936d210bda465d7ffd000000000000000000000000d86eefc60752ff29549f19b54da42523b197b276. got: 0x7ef8f8a091df96f7eaa733e24c5e7d66950b2c81f52458678e878b53b8f83a6eb0f4820294deaddeaddeaddeaddeaddeaddeaddeaddead00019442000000000000000000000000000000000000158080830f424080b8a4440a5e20000f42400000000000000000000000000000000066ee84a40000000002a0565900000000000000000000000000000000000000000000000000000002540be400000000000000000000000000000000000000000000000000000000000000000117ab1602193cd09273e9149282142da39d923e59c5584d1c4d801c1d4be3d9f7000000000000000000000000d86eefc60752ff29549f19b54da42523b197b276" unsafe=0x1d8f3f6139ea6238e75bf79444a98f68ee5ddab78015058b33f37d3c71ec6638:2381876 pending_safe=0x44a9e38be2197b09dce4f5b52bedc6ae0f4f2805ab467e3b2ebfd3f7a507e3a2:2345040 safe=0x44a9e38be2197b09dce4f5b52bedc6ae0f4f2805ab467e3b2ebfd3f7a507e3a2:2345040
t=2024-09-21T20:32:53+0000 lvl=warn msg="did not finish previous block building, starting new building now" prev_onto=0x1d8f3f6139ea6238e75bf79444a98f68ee5ddab78015058b33f37d3c71ec6638:2381876 prev_payload_id=0x747b3cffdb20b41e new_onto=0x44a9e38be2197b09dce4f5b52bedc6ae0f4f2805ab467e3b2ebfd3f7a507e3a2:2345040
t=2024-09-21T20:32:53+0000 lvl=warn msg="engine is building block that reorgs previous unsafe head" onto=0x44a9e38be2197b09dce4f5b52bedc6ae0f4f2805ab467e3b2ebfd3f7a507e3a2:2345040 unsafe=0x1d8f3f6139ea6238e75bf79444a98f68ee5ddab78015058b33f37d3c71ec6638:2381876
t=2024-09-21T20:32:55+0000 lvl=info msg="inserted block" hash=0xa88636cea1b115bfef8c14c2bd5c684ecca7ee9bba183a9ef28eee40d2f542a8 number=2345041 state_root=0x19243a9f7e519cabd8f4e842aea431a9a7ff5c7eee689313edbfbc111f3342bb timestamp=1726907556 parent=0x44a9e38be2197b09dce4f5b52bedc6ae0f4f2805ab467e3b2ebfd3f7a507e3a2 prev_randao=0x0000000000000000000000000000000000000000000000000000000000000000 fee_recipient=0x4200000000000000000000000000000000000011 txs=1 update_safe=true
t=2024-09-21T20:32:55+0000 lvl=info msg="Sync progress" reason="processed safe block derived from L1" l2_finalized=0x483e1b4e3a8a66cd71005c2dda9b324a98641fe259ac573bd7d07a1f7462a39e:2344983 l2_safe=0xa88636cea1b115bfef8c14c2bd5c684ecca7ee9bba183a9ef28eee40d2f542a8:2345041 l2_pending_safe=0xa88636cea1b115bfef8c14c2bd5c684ecca7ee9bba183a9ef28eee40d2f542a8:2345041 l2_unsafe=0xa88636cea1b115bfef8c14c2bd5c684ecca7ee9bba183a9ef28eee40d2f542a8:2345041 l2_backup_unsafe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_time=1726907556 l1_derived=0x46ead0121f93fc10e5ed4809c9515c77ef9609a48e1b94b7cf160d371550a223:44076697
t=2024-09-21T20:32:55+0000 lvl=warn msg="Fetch receipts cache missed when sequencer building block" current=0x17ab1602193cd09273e9149282142da39d923e59c5584d1c4d801c1d4be3d9f7:44062297 current_time=1726907556 l2_head=0xa88636cea1b115bfef8c14c2bd5c684ecca7ee9bba183a9ef28eee40d2f542a8:2345041 l2_head_time=1726907556
t=2024-09-21T20:32:55+0000 lvl=info msg="creating new block" parent=0xa88636cea1b115bfef8c14c2bd5c684ecca7ee9bba183a9ef28eee40d2f542a8:2345041 l1Origin=0x17ab1602193cd09273e9149282142da39d923e59c5584d1c4d801c1d4be3d9f7:44062297
t=2024-09-21T20:32:55+0000 lvl=info msg="sequencer started building new block" payload_id=0x804b38f07150b251 l2_parent_block=0xa88636cea1b115bfef8c14c2bd5c684ecca7ee9bba183a9ef28eee40d2f542a8:2345041 l2_parent_block_time=1726907556
t=2024-09-21T20:32:55+0000 lvl=info msg="Generating next batch" epoch=0x17ab1602193cd09273e9149282142da39d923e59c5584d1c4d801c1d4be3d9f7:44062297 timestamp=1726907557
t=2024-09-21T20:32:55+0000 lvl=info msg="generated attributes in payload queue" txs=1 timestamp=1726907557
t=2024-09-21T20:32:55+0000 lvl=warn msg="did not finish previous block building, starting new building now" prev_onto=0xa88636cea1b115bfef8c14c2bd5c684ecca7ee9bba183a9ef28eee40d2f542a8:2345041 prev_payload_id=0x804b38f07150b251 new_onto=0xa88636cea1b115bfef8c14c2bd5c684ecca7ee9bba183a9ef28eee40d2f542a8:2345041
t=2024-09-21T20:32:55+0000 lvl=info msg="inserted block" hash=0x41a61a1fe1ace9dda4d95d1498e3ab2454a46a56f306eaad16d3da8386f98e19 number=2345042 state_root=0xa338eae65f39626cfb24c98a486b589baa594d07dad25bddf0a4c2ad006ea9ec timestamp=1726907557 parent=0xa88636cea1b115bfef8c14c2bd5c684ecca7ee9bba183a9ef28eee40d2f542a8 prev_randao=0x0000000000000000000000000000000000000000000000000000000000000000 fee_recipient=0x4200000000000000000000000000000000000011 txs=1 update_safe=true
t=2024-09-21T20:32:55+0000 lvl=info msg="Sync progress" reason="processed safe block derived from L1" l2_finalized=0x483e1b4e3a8a66cd71005c2dda9b324a98641fe259ac573bd7d07a1f7462a39e:2344983 l2_safe=0x41a61a1fe1ace9dda4d95d1498e3ab2454a46a56f306eaad16d3da8386f98e19:2345042 l2_pending_safe=0x41a61a1fe1ace9dda4d95d1498e3ab2454a46a56f306eaad16d3da8386f98e19:2345042 l2_unsafe=0x41a61a1fe1ace9dda4d95d1498e3ab2454a46a56f306eaad16d3da8386f98e19:2345042 l2_backup_unsafe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_time=1726907557 l1_derived=0x46ead0121f93fc10e5ed4809c9515c77ef9609a48e1b94b7cf160d371550a223:44076697

[STdevK]

The team is looking into it

[Eoous]

For a while batcher failed to submit batches for having no bnb.

After recharging batcher address, op-node shows the log.

We found the problem due to batcher having no bnb, so i think if batcher used all bnb would trige the problem.

And even stopping batcher, currentBlockNumber(means finalized block number) in op-proposer also increases

[krish-nr]

For a while batcher failed to submit batches for having no bnb.

After recharging batcher address, op-node shows the log.

We found the problem due to batcher having no bnb, so i think if batcher used all bnb would trige the problem.

And even stopping batcher, currentBlockNumber(means finalized block number) in op-proposer also increases

How long did the batcher stop submitting to L1 due to insufficient balance?

[Eoous]

For a while batcher failed to submit batches for having no bnb.
After recharging batcher address, op-node shows the log.
We found the problem due to batcher having no bnb, so i think if batcher used all bnb would trige the problem.
And even stopping batcher, currentBlockNumber(means finalized block number) in op-proposer also increases

How long did the batcher stop submitting to L1 due to insufficient balance?

4 hours

[krish-nr]

For a while batcher failed to submit batches for having no bnb.
After recharging batcher address, op-node shows the log.
We found the problem due to batcher having no bnb, so i think if batcher used all bnb would trige the problem.
And even stopping batcher, currentBlockNumber(means finalized block number) in op-proposer also increases

How long did the batcher stop submitting to L1 due to insufficient balance?

4 hours

what's the config of your SequencerWindowSize? From the log it seems unsafe:2381876 and pending safe:2345040 gap is about 10 hours, If SequencerWindowSize is less than this, it may cause a fork leading to a mismatch

[Eoous]

SequencerWindowSize is 14400

[Eoous]

For a while batcher failed to submit batches for having no bnb.
After recharging batcher address, op-node shows the log.
We found the problem due to batcher having no bnb, so i think if batcher used all bnb would trige the problem.
And even stopping batcher, currentBlockNumber(means finalized block number) in op-proposer also increases

How long did the batcher stop submitting to L1 due to insufficient balance?

4 hours

what's the config of your SequencerWindowSize? From the log it seems unsafe:2381876 and pending safe:2345040 gap is about 10 hours, If SequencerWindowSize is less than this, it may cause a fork leading to a mismatch

Can i fix it? SequencerWindowSize doesn't effect safe and finalized, right?

For the problem, safe and finalized increase without op-batch submitting