Monero Research Lab Meeting - Wed 05 June 2024, 17:00 UTC

[Rucknium]

Location: Libera.chat, #monero-research-lab | Matrix

Join the Monero Matrix server if you don't already have a Matrix account.

Time: 17:00 UTC Check in your timezone

Main discussion topics:

1. Greetings

2. Updates. What is everyone working on?

3. Stress testing monerod

4. Potential measures against a black marble attack.

5. Research Pre-Seraphis Full-Chain Membership Proofs.

6. Any other business

7. Confirm next meeting agenda

Please comment on GitHub in advance of the meeting if you would like to propose an agenda item.

Logs will be posted here after the meeting.

Meeting chairperson: Rucknium

Previous meeting agenda/logs:

#1015

[Rucknium]

Logs

< 0​xfffc:monero.social > Hi everyone

< a​rticmine:monero.social > Hi

< r​ucknium:monero.social > Meeting time! #1020

< r​ucknium:monero.social > 1) Greetings

< rbrunner > Hello

< jberman > hello

< spackle > hi

< r​ucknium:monero.social > 2) Updates. What is everyone working on?

< c​haser:monero.social > hello

< v​tnerd:monero.social > Hi

< jberman > me: trim_tree for the fcmp tree

< r​ucknium:monero.social > me: Working a little more on the black marble optimal ring size and fee analysis. Helping spackle with setting up the new stressnet. Working on my MoneroKon presentation "Hard Data on Banking the Unbanked with Cryptocurrency".

< a​rticmine:monero.social > I am finalizing the scaling changes. Will be presenting them at MoneroKon, on Friday.

< b​oog900:monero.social > Hi

< v​tnerd:monero.social > Me: testing the LWS remote scanning feature, which led to a few bug fixes

< v​tnerd:monero.social > Although the bugs were unrelated to the feature, and required some backporting to release branches

< 0​xfffc:monero.social > me: worked on few reviews here and there. Worked on a fix for duplicate transactions in fluff queue. Had no familiarity with fluff and stem, so took a long time to understand the code, and finally it is done. Initial planning with spackle and rucknium on monero stressnet and monerod torture test.

< r​ucknium:monero.social > 3) Stress testing monerod monero-project/monero#9348

< spackle > me: running a testnet fork for stress testing

< r​ucknium:monero.social > spackle, do you want to explain what you've been working on?

< spackle > I've made a few attempts at stress testing monerod

< spackle > Running a 10 node private testnet on a single machine did not stress things as desired.

< spackle > So I have forked the testnet to create an abusable network for testing monerod.

< spackle > The 'stressnet' is running now (https://github.com/spackle-xmr/monero)

< spackle > The next important step is creating a release + binaries for others to run. Gitian is new to me, and I am having some difficulties with the process.

< r​ucknium:monero.social > Recently we got the txpool to almost 200MB. More than 16 hours of backlogged txs.

< spackle > That said, things are running fine and I have been able to see some of the limits of the daemon with some early spamming. Even with low connection counts, performance limits have been observed.

< r​ucknium:monero.social > The monerod with the v17 hardfork for testnet self-compiles and runs fine. The harder part is trying to have a real build process for the binaries on all the operating systems so people who can't/won't compile can join the network if they want.

< r​ucknium:monero.social > I think we saw problems with tx and block propagation already with just 6 nodes.

< r​ucknium:monero.social > plowsof and selsta have helped with trying to troubleshoot the gitian build process :)

< spackle > Which is much appreciated, of course.

< r​ucknium:monero.social > I think 0xfffc is going to help with how to measure performance precisely and try to figure out where in the code the bottlenecks are. I think what we have is a workable base.

< sech1 > is there any write up of the problems? What was the bottleneck - CPU, memory, network, SSD?

< 0​xfffc:monero.social > Yes. Once we got the stressnet, the other side (debugging/profiling) will be my responsibility

< 0​xfffc:monero.social > That yes was reply to this message.

< r​ucknium:monero.social > sech1: I think at this stage we are just experimenting and making sure that the network is running correctly and the spam script works. And figuring out the rough limits.

< r​ucknium:monero.social > spackle's node that received the spam got to 17GB of RAM at one point. Before we had a hypothesis that maybe monerod's LMDB was loading the DB into RAM and that's why some people were seeing too much RAM usage. But the testnet DB is less than 10 GB, so that hypothesis seems unlikely now.,

< r​ucknium:monero.social > "A lot of 150/2 transactions in the txpool causes memory spike / OOM daemon" monero-project/monero#9317

< b​oog900:monero.social > I made a small program using Cuprates P2P stack to make and maintain loads of connections to a single node, my plan is to pop blocks back to around when nodes crashed and start pushing txs from the blocks after to the txpool

< r​ucknium:monero.social > My hypothesis is that something about preparing txs to be sent p2p and to wallets uses RAM, and then the RAM isn't released maybe.

< b​oog900:monero.social > the connections don't do anything, just enough to stay connected, but monerod will still fluff txs to them

< a​rticmine:monero.social > What kind of TPS are we talking about here?

< r​ucknium:monero.social > I think spackle was pushing 5 tx/sec. It was pushing up txpool size since blocks can't include that many txs at that rate of course

< spackle > For the stressnet? In my trials I have used a single daemon instance with an rpc wallet. That setup is limited to creating ~15tps, and I believe that entirely occupies the daemon.

< 0​xfffc:monero.social > Once we had stressnet (which is very useful idea), it is going to be much easier to debug the monerod code and find bottlenecks. I would focus on getting stressnet going as first step. ( Thanks to spackle and ruck )

< r​ucknium:monero.social > spackle seems to have a configurable spammer. It can change the tx/sec and the fee of the txs

< spackle > During tx creation the daemon connected to the rpc wallet would not be able to send txs to other nodes unless there was a break in the spam. At times, there was 40MB of txs that were not relayed.

< r​ucknium:monero.social > Anyone can join the stressnet by compiling this monerod and running as --testnet: https://github.com/spackle-xmr/monero . Ignore the releases because we are still not getting the releases right.

< r​ucknium:monero.social > --max-connections-per-ip=1000 is also recommended since there are a few nodes running from the same IP

< v​tnerd:monero.social > Rucknium: there's a couple of vectors and deque where shrink to fit could be used. Seems unlikely but worth a look

< v​tnerd:monero.social > Or no deque, just a vector I think

< jberman > sounding like a productive endeavor, thank you guys

< 0​xfffc:monero.social > One of the areas that needs that stressnet is locking. My rwlock, initially introduces something like %15 speed up compared to baseline, but later it causes slowdown of %10 compared to baseline. (As sgp tested it under their heavy usage node). Once we had stressnet, I can run it long time and find out the reason for slowdown for rwlock. Right now I don’t have access to any kind

< 0​xfffc:monero.social > of heavy usage node.

< a​rticmine:monero.social > I agree. This is a very productive endeavor. Thank you.

< r​ucknium:monero.social > Special thanks to spackle!

< r​ucknium:monero.social > 4) Potential measures against a black marble attack monero-project/research-lab#119

< r​ucknium:monero.social > I write two more solution concepts for my model. The solution concept in the current paper draft is the ring size/fee optimizers of cost effectiveness when you have certain inequality constraints. Two other ones: 1) Best cost effectiveness when you have the equality constraint of a certain effective ring size (i.e. must be on the effective ring size line). 2) Give Alice a budget<clipped message

< r​ucknium:monero.social > constraint. Alice has to spend less than a certain amount on aggregate tx fees and node storage costs. That's another inequality constraint to the main model. I don't have results to share about those two solution concepts yet.

< c​haser:monero.social > great! I don't know how helpful that is, but I think a constraint on the tx generation/verification time would (also) be useful

< r​ucknium:monero.social > That could be a good idea, too. Thanks.

< r​ucknium:monero.social > 5) Research Pre-Seraphis Full-Chain Membership Proofs. https://www.getmonero.org/2024/04/27/fcmps.html

< r​ucknium:monero.social > Any updates on FCMP to discuss?

< jberman > no news from me

< r​ucknium:monero.social > Any other agenda items?

< r​ucknium:monero.social > I have a question maybe someone could answer. When exactly does monerod verify the expensive cryptography in a tx? When nodes get a new tx from peers, there is some verification, but from my view of the code it looks like it may be quick hash verification. But probably nodes do not wait until they get a fluffy block to do the full verification.

< r​ucknium:monero.social > To me it looks like nodes do the hash verification on every tx that they get from a peer even if they have seen the tx from another peer before. If they are doing the full verification at that step, that's more reason to consider boog900 's suggestion to reduce duplicate tx gossip messages.

< r​ucknium:monero.social > This log message fires whenever nodes get new txs from peers: https://github.com/monero-project/monero/blob/c8214782fb2a769c57382a999eaf099691c836e7/src/cryptonote_protocol/cryptonote_protocol_handler.inl#L990

< r​ucknium:monero.social > Which seems to run this hash verification function: https://github.com/monero-project/monero/blob/c8214782fb2a769c57382a999eaf099691c836e7/src/cryptonote_basic/cryptonote_format_utils.cpp#L253

< jberman > in here: https://github.com/monero-project/monero/blob/c8214782fb2a769c57382a999eaf099691c836e7/src/cryptonote_protocol/cryptonote_protocol_handler.inl#L1038

< r​ucknium:monero.social > I have done most of the parsing work of the p2p logs that were running during some of the spam. This question formed when I was looking at the data since I get all these messages about each tx being added (to something).

< r​ucknium:monero.social > Thanks!

< r​ucknium:monero.social > We can end the meeting here. Thanks everyone.

< 0​xfffc:monero.social > Thanks everyone

< c​haser:monero.social > thank you all for your work!

< v​tnerd:monero.social > https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_core/tx_pool.cpp#L263

< v​tnerd:monero.social > I believe is where the expensive crypto checks are performed. This will only occur if the tx isn't in the txpool already

< v​tnerd:monero.social > monero-project/monero#9135 is a review that should speed up new tx processing a bit. A reminder (mainly for myself) to review as this could/should be shipped by now