Recover from backend timeout

[bota87]

If the connection to bitcoind fails at some point (for a bitcoind restart for example) than jam can't reconnect and it's needed to restart it.
Verified with this image ghcr.io/joinmarket-webui/jam-standalone:v0.1.5-clientserver-v0.9.9

Remove leftover wallet lockfiles before startup...
Waiting for bitcoind to accept RPC requests...
Successfully waited for bitcoind to accept RPC requests.
Creating wallet jm_webui_default if missing...
Loading wallet jm_webui_default...
[  OK  ] nginx
[  OK  ] tor
[  OK  ] ob-watcher
[  OK  ] jmwalletd
[  OK  ] boot
dinit: Service jmwalletd process terminated with exit code 1
dinit: Service jmwalletd process terminated with exit code 1
dinit: Service jmwalletd process terminated with exit code 1
dinit: Service jmwalletd restarting too quickly; stopping.
dinit: Service jmwalletd restarting too quickly; stopping.
[STOPPD] jmwalletd

[theborakompanioni]

Hey @bota87, thanks for the report!

We have a couple of parameters for dinit services to control that. Currently, the defaults are used, which seem to be not applicable. (default: restart-limit-count := 3, restart-delay := 200ms)

[restart-delay = XXX.YYYY](https://davmac.org/projects/dinit/man-pages-html/dinit-service.5.html#restart~2)
    Specifies the minimum time (in seconds) between automatic restarts. Enforcing a sensible minimum prevents Dinit from consuming a large number of process cycles in case a process continuously fails immediately after it is started. The default is 0.2 (200 milliseconds).
[restart-limit-count = NNN](https://davmac.org/projects/dinit/man-pages-html/dinit-service.5.html#restart~4)
    Specifies the maximum number of times that a service can automatically restart over the interval specified by restart-limit-interval. Specify a value of 0 to disable the restart limit. The default value is 3.

I'd suggest restart-limit-count := 42 and restart-delay := 10s which means 7min of downtime window. What do you think? Do you have other values in mind?

PS: I'll transfer this issue to the jam-docker repo soon.

[bota87]

The suggested parameters looks good to cover a backend restart, but what happen if the downtime last longer than 7 minutes? The app enter in an unrecoverable state like now? It's no better to kill the process and let the container restart policy do it's work?

[theborakompanioni]

The suggested parameters looks good to cover a backend restart, but what happen if the downtime last longer than 7 minutes? The app enter in an unrecoverable state like now? It's no better to kill the process and let the container restart policy do it's work?

Yes, then it would require a restart of the container..
Letting the container restart itself is a good idea. However, I am not totally sure how to do that with the current supervisor setup..
What would be easily possible is to restart services without any limitations, i.e. restarting indefinitely..

[bota87]

That's ok for me, it's better than the current behavior