changefeedccl: core changefeed fails in high data volume throughput test

[asg0451]

I observed and reproduced a core changefeed fail with a batch timestamp error while testing something unrelated.

Reproduction:

Given ./cockroach start-single-node --insecure running.

In 2 terminals, A and B:

A (in ./cockroach sql --insecure):

create table stuff2 (a int primary key);

B:

./cockroach sql --insecure -e "EXPERIMENTAL CHANGEFEED FOR TABLE stuff2 WITH OPTIONS (updated, resolved)" | grep -v stuff2 # start a changefeed but discard the data so it doesnt spam your terminal

A:

insert into stuff2 (a)  select i from generate_series(0, 10000) as g(i);
insert into stuff2 (a)  select i from generate_series(10001, 50000) as g(i);
insert into stuff2 (a)  select i from generate_series(50001, 100000) as g(i);
insert into stuff2 (a)  select i from generate_series(100001, 500000) as g(i);
insert into stuff2 (a)  select i from generate_series(500001, 1000000) as g(i);
insert into stuff2 (a)  select i from generate_series(1000001, 5000000) as g(i);
insert into stuff2 (a)  select i from generate_series(5000001, 10000000) as g(i);

..And then wait.. after a minute or so, the core changefeed in B will fail with:

(error encountered after some results were delivered)
ERROR: batch timestamp 1730910286.954309999,0 must be after replica GC threshold 1730910298.890121000,0 (r140: /Table/105/4/{3384343-5848963})

(table 105 here is stuff2; you can check that with SHOW RANGES FROM DATABASE defaultdb WITH TABLES)

[blathers-crl]

Hi @asg0451, please add branch-* labels to identify which branch(es) this C-bug affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

cc @cockroachdb/cdc