Event loop is regularly being blocked for longish periods

[ulope]

Problem Definition

Using the gevent monitoring thread (enable by setting the env var GEVENT_MONITOR_THREAD_ENABLE=1) shows that we're regularly blocking the event loop for longish periods (the max allowed blocking time can be controlled via GEVENT_MAX_BLOCKING_TIME, I used 0.1 and 0.5 as an example).

The main blocking culprits seem to be:

• sqlite
• deepcopy calls inside the state machine

Possible solutions

For sqlite there is gsqlite3 which on paper sounds like a solution.

For the state machine we should look into using persistent / functional datastructures as a way to avoid having to use deepcopy. Pyrsistent looks promising.

[rakanalh]

The main blocking culprits seem to be:

• sqlite
• deepcopy calls inside the state machine

Is there a way to construct a flame graph to support this?

[hackaugusto]

https://github.com/uber/pyflame

[hackaugusto]

Actually, I tried pyflame with py3.7 and py3.6 and it doesn't work with either.

For py3.7 the abi is not supported.
For py3.6 It seems to be a known bug uber-archive/pyflame#129 .
For py3.5 raiden is not compatible

I may reintroduce one of the tools we had for profilling

[ulope]

Is there a way to construct a flame graph to support this?

Gevent already shows where the blocking occurs when running with the variables I mentioned above.
It just needs someone taking the time to look through the generated output.
To make this a bit easier the exception_stream variable can be set to a file for example so output isn't intermingled with the raiden logs.

For profiling there is also py-spy (which claims to support 3.3. - 3.7).
The gevent docs point to https://github.com/nylas/nylas-perftools

[hackaugusto]

We have done a lot of work on this issue, since blocking the event loop results in presence problems with the matrix transport, which led to transfer failures because the nodes appeared to be offline.

There are still some longish periods of time were the loop may block, however, they are not a source of problem ATM. The next step should be to add benchmarks and create issues based on that. For now I'm closing this issue.