mempool wont sync

[minioff]

In certain conditions I are observing mempool fails to sync from remote bitcoind. The setup is

bitcoind on machine A
fulcrum on machine A (connected to local bitcoind on A)
fulcrum on machine B (connected to remote bitcoind on A)

• The issue can only be observed on fulcrum running on remote instance B.
• Machine B is located on the other side of the globe about 200ms apart. Both machines are in data centers.
• Both fulcrum are running same versions and configurations.

The conditions to observe failure are a larger mempool (eg 100mb and above) and somewhat slow connection to remote bitcoind.

Working configuration with mempool decreased to 50mb I can observe

<Controller> 227 mempool txs involving 1162 addresses
<SynchMempool> (Debug) downloaded 292 txs (failed: 0, ignored: 0), elapsed so far: 43.480 secs
<SynchMempool> (Debug) Mempool: pre-cache of 405 confirmed spends from db took 19.351 msec
<SynchMempool> (Debug) 519 mempool txs involving 1852 addresses (exclusive lock held for 1.859 msec)
<Controller> (Debug) SynchMempool elapsed total: 43.502 secs
....

<Controller> 712 mempool txs involving 2519 addresses
<SynchMempool> (Debug) downloaded 257 txs (failed: 0, ignored: 0), elapsed so far: 38.162 secs
<SynchMempool> (Debug) Mempool: pre-cache of 182 confirmed spends from db took 22.836 msec
<SynchMempool> (Debug) 969 mempool txs involving 2884 addresses (exclusive lock held for 1.680 msec)
<Controller> (Debug) SynchMempool elapsed total: 38.188 secs

With mempool at 100 mb only this line is shown in the log

<SynchMempool> (Debug) getrawmempool: got reply with 8077 items, 0 ignored, 0 dropped, 8077 new

No further output regarding mempool operations is seen in the log. Mempool always remains empty.

./FulcrumAdmin -p 8000 getinfo | jq .mempool
{
  "addresses": 0,
  "avg_fee_sats_B": 0,
  "size_bytes": 0,
  "txs": 0
}

Fulcrum is running with --polltime 30 --bd-timeout 30 .

[minioff]

I have been trying to play with high values (above 100) for bd-clients in fulcrum and rpcthreads in bitcoind respectively, but it doesn't seem to be making a difference. Can see in bitcoind log with -debug=rpc that the (network) throughput is the issue. Is getrawtransaction call parallelized?

[cculianu]

Fulcrum is not designed to run in this way, with 1 machine so far away from another. This is because of limitations in how bitcoind RPC provides data. As a result, it is impossible to synch a full mempool due to the latency. Actually it synchs but I reckon it would take hours.

<SynchMempool> (Debug) downloaded 292 txs (failed: 0, ignored: 0), elapsed so far: 43.480 secs

It took 43 seconds to only download 292 txs. This is about ~6.79 txs/second which is on the order of your observed 200msec of latency (actually a little bit better than 200 msec per tx, at 150 msec per tx).

Fulcrum must do 1 RPC round-trip requesting the txn for EACH tx in the mempool. If you do the math, this means that for a 100k txn mempool (or whatever ridiculous thing is going on now on BTC) if we assume ~150msec per tx, it would take over 4 hours to synch a full mempool.

This is because there is no way for Fulcrum to retrieve the entire raw mempool in 1 download. The calls that bitcoind RPC provides do not allow for this. It must do a round-trip to bitcoin each time, issuing a getrawtransaction call, for each mempool entry.

[cculianu]

Is getrawtransaction call parallelized?

Not in Fulcrum.. if it were perhaps things would be #-of-threads faster. This would perhaps be a measure to speed it up but it's still impractical. Default bitcoin has an rpc work queue of 16. So even if you did 16 simultaneous calls -- it would still take 15 minutes to synch 1 very full mempool on your setup.

And that's assuming NO OTHER client asks bitcoind for stuff. If it does, bitcoind returns an error message when the work queue is full saying, basically, "sorry work queue full". So you can't even use all 16 work queue slots in this way for just mempool. You'd need to leave 4-10 slots free so we are looking at like 30 mins + just to synch the mempool initially.

Basically -- Fulcrum was not designed to work in such a high latency environment and it's impractical. Yes, parallel calls to getrawtransaction may help (to a point) since you can pay the fixed latency cost of 150 msec in chunks of 16 txns, say -- but it's still impractical.

Fulcrum should really be run with a local bitcoind instance.

[minioff]

Thanks for confirming my assumptions and observations.

Parallel getrawtx would indeed solve this issue as the machines can easily support 100+ threads.

[cculianu]

Yeah it probably would, esp with a 100x speedup. I guess I just don't have the resources right now to devote to working on this. It is solvable, for sure, with parallelism. It would be quite an effort and for a niche case, for basically a "free" open source app, kind of too much work for me to do. :)