Indy Node crashes and closes its listener on heavy load or load for longer period

[SuryaSHalwasia]

I am using Indy Node in Ubuntu 20 outside docker. In docker when I run the load test it doesn't crash as it does in Ubuntu 20.
This is the error it gives me

2024-07-05 17:19:08,181|WARNING|stacks.py|Node4C: connections number fell below the limit! Actual: 347, limit: 400.
2024-07-05 17:19:08,181|WARNING|stacks.py|Node4C: client stack restart is not needed anymore.
2024-07-05 17:19:08,878|WARNING|stacks.py|Node4C: connections limit reached! Actual: 410, limit: 400.
2024-07-05 17:19:08,878|WARNING|stacks.py|Node4C: connections limit reached but too few time spent since client stack start, restart it later.
2024-07-05 17:19:15,063|INFO|ordering_service.py|Node4:1 set last ordered as (0, 21)
2024-07-05 17:19:15,071|INFO|ordering_service.py|Node4:1 ordered batch request, view no 0, ppSeqNo 21, ledger 1, state root None, txn root None, audit root None, requests ordered 532, discarded 0
2024-07-05 17:19:15,075|INFO|ordering_service.py|Node4:1 set last ordered as (0, 22)
2024-07-05 17:19:15,081|INFO|ordering_service.py|Node4:1 ordered batch request, view no 0, ppSeqNo 22, ledger 1, state root None, txn root None, audit root None, requests ordered 182, discarded 0
2024-07-05 17:19:15,081|INFO|ordering_service.py|Node4:1 set last ordered as (0, 23)
2024-07-05 17:19:15,086|INFO|ordering_service.py|Node4:1 ordered batch request, view no 0, ppSeqNo 23, ledger 1, state root None, txn root None, audit root None, requests ordered 473, discarded 0
2024-07-05 17:19:15,768|INFO|ordering_service.py|Node4:1 set last ordered as (0, 24)
2024-07-05 17:19:15,769|INFO|ordering_service.py|Node4:1 ordered batch request, view no 0, ppSeqNo 24, ledger 1, state root None, txn root None, audit root None, requests ordered 37, discarded 0
2024-07-05 17:19:24,202|INFO|ordering_service.py|Node4:0 set last ordered as (0, 16)
2024-07-05 17:19:25,005|INFO|ordering_service.py|Node4:0 ordered batch request, view no 0, ppSeqNo 16, ledger 1, state root J3kBWhVEkrhbtwhw7tuB1Y78o49JMSfoUebFvX96xAcc, txn root B9TH7NwF4M6fw6Wk6qYJRBX1VTJjnvKJZPfTSCFGiGPU, audit root 56dnDyFMkxPDTr5xNpTvzfbYFPhhdkDePSGjz5TBuxer, requests ordered 1000, discarded 0
2024-07-05 17:19:33,007|NOTIFICATION|looper.py|Looper shutting down now...
2024-07-05 17:19:33,007|INFO|motor.py|Node4 changing status from started to stopping
2024-07-05 17:19:33,007|INFO|node.py|node Node4 current stats

node inbox size : 0
client inbox size : 0
age (seconds) : 155.4772744178772
next check for reconnect: 6.888445964999988
node connections : {'Node3', 'Node2', 'Node1'}
f : 1
master instance : 0
replicas : 2
view no : 0
rank : 3
msgs to replicas : 7
action queue : 0 140633492501984
action queue stash : 4 140633492502080
2024-07-05 17:19:33,008|INFO|node.py|Node4 reseting...
2024-07-05 17:19:33,011|NOTIFICATION|zstack.py|stack Node4 closing its listener
2024-07-05 17:19:33,012|NOTIFICATION|zstack.py|stack Node4 stopped
2024-07-05 17:19:33,012|NOTIFICATION|zstack.py|stack Node4C closing its listener

And all the nodes crash. My specs are quite good in each of the ubuntu instance. 16GB RAM and 8 CPU.
The dockerized version of indy node is half of it and doesn't crash like this.
What should I do? Thanks

[WadeBarnes]

It doesn't look like the node(s) are crashing. It looks more like the indy-node-control service is restarting them (possibly due to performance while under load). The indy-node-control service does not run in the docker containers.

Could you please provide the following information (redact any private info):

/var/log/indy/<network_name>/<node_alias>.log
/var/log/indy/node_control.log
sudo journalctl -u indy-node
sudo journalctl -u indy-node-control
sudo validator-info -v
cat /etc/indy/indy_config.py
cat /etc/indy/indy.env
cat /etc/indy/node_control.conf

[SuryaSHalwasia]

So here are the details
In /var/log/indy/sandbox/Node1.log
2024-07-07 23:36:28,927|INFO|ordering_service.py|Node1:0 ordered batch request, view no 0, ppSeqNo 11, ledger 1, state root DfuSUSvHxfsAmBHE1sj9anpaa8vsMDfRx9KbxqMSneFS, txn root BVnQVtMdG7chJR8M2Ge6s4GRaoPXkzHpbEyTDeLw2ypB, audit root 5R9kkYBz3xJDrAet4DYchGWpmxWNdJwYEN1nG8smQX9V, requests ordered 783, discarded 0
2024-07-07 23:36:35,646|NOTIFICATION|looper.py|Looper shutting down now...
2024-07-07 23:36:35,646|INFO|motor.py|Node1 changing status from started to stopping
2024-07-07 23:36:35,646|INFO|node.py|node Node1 current stats

node inbox size : 0
client inbox size : 0
age (seconds) : 352.6427562236786
next check for reconnect: 2.526324950000344
node connections : {'Node2', 'Node3', 'Node4'}
f : 1
master instance : 0
replicas : 2
view no : 0
rank : 0
msgs to replicas : 0
action queue : 0 139717908141536
action queue stash : 5 139717908141632
2024-07-07 23:36:35,646|INFO|node.py|Node1 reseting...
2024-07-07 23:36:35,660|NOTIFICATION|zstack.py|stack Node1 closing its listener
2024-07-07 23:36:35,660|NOTIFICATION|zstack.py|stack Node1 stopped
2024-07-07 23:36:35,660|NOTIFICATION|zstack.py|stack Node1C closing its listener
2024-07-07 23:36:35,664|NOTIFICATION|zstack.py|stack Node1C stopped
2024-07-07 23:36:35,665|INFO|node.py|Node1 closing key-value storages
2024-07-07 23:36:36,658|INFO|motor.py|Node1 changing status from stopping to stopped
2024-07-07 23:36:36,658|NOTIFICATION|looper.py|Looper shut down in 1.012 seconds.

In others it is
2024-07-07 23:36:35,767|NOTIFICATION|keep_in_touch.py|Node3's connections changed from {'Node2', 'Node4', 'Node1'} to {'Node2', 'Node4'}
2024-07-07 23:36:35,767|NOTIFICATION|keep_in_touch.py|CONNECTION: Node3 disconnected from Node1

2024-07-07 23:36:35,767|INFO|motor.py|Node3 changing status from started to started_hungry
2024-07-07 23:36:35,767|NOTIFICATION|primary_connection_monitor_service.py|Node3:0 lost connection to primary of master
2024-07-07 23:36:35,767|INFO|primary_connection_monitor_service.py|Node3:0 scheduling primary connection check in 60 sec

In /var/log/indy/node_control.log (This was only present in Node1)
2024-06-26 20:15:24,574|INFO|node_control_tool.py|Node control tool is starting up on localhost port 30003
2024-06-26 20:15:25,142|INFO|node_control_utils.py|Successfully put indy-plenum indy-node python3-ioflo python3-orderedset python3-base58 python3-prompt-toolkit python3-rlp python3-sha3 python3-libnacl python3-six python3-portalocker python3-sortedcontainers python3-dateutil python3-semver python3-psutil python3-pyzmq python3-jsonpickle python3-rocksdb python3-pympler python3-packaging python3-timeout-decorator python3-distro packages on hold
2024-06-26 20:15:25,142|DEBUG|node_control_tool.py|Waiting for the next event

For journalctl -u indy-node
-- Logs begin at Sat 2023-12-02 18:58:18 UTC, end at Mon 2024-07-08 14:56:20 UTC. --
-- No entries --

For indy-node-control

-- Logs begin at Sat 2023-12-02 18:58:18 UTC, end at Mon 2024-07-08 14:58:55 UTC. --
Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: Started Service for upgrade of existing Indy Node and other operations.
Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: Stopping Service for upgrade of existing Indy Node and other operations...
Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: indy-node-control.service: Succeeded.
Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: Stopped Service for upgrade of existing Indy Node and other operations.
Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: Started Service for upgrade of existing Indy Node and other operations.
Jun 26 20:15:24 ip-172-31-40-65 env[19720]: 2024-06-26 20:15:24,574|INFO|node_control_tool.py|Node control tool is starting up o>
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: indy-plenum set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: indy-node set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-ioflo set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-orderedset set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-base58 set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-prompt-toolkit set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-rlp set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-sha3 set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-libnacl set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-six set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-portalocker set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-sortedcontainers set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-dateutil set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-semver set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-psutil set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-pyzmq set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-jsonpickle set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-rocksdb set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-pympler set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-packaging set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-timeout-decorator set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-distro set on hold.
Jun 26 20:15:25 ip-172-31-40-65 env[19720]: 2024-06-26 20:15:25,142|INFO|node_control_utils.py|Successfully put indy-plenum indy>
Jun 26 20:15:25 ip-172-31-40-65 env[19720]: 2024-06-26 20:15:25,142|DEBUG|node_control_tool.py|Waiting for the next event
Jun 27 04:22:39 ip-172-31-40-65 systemd[1]: Stopping Service for upgrade of existing Indy Node and other operations...
Jun 27 04:22:40 ip-172-31-40-65 systemd[1]: indy-node-control.service: Succeeded.
Jun 27 04:22:40 ip-172-31-40-65 systemd[1]: Stopped Service for upgrade of existing Indy Node and other operations.

For validator-info
"Hardware":
"HDD_all": 6405 Mbs
"RAM_all_free": 31119 Mbs
"RAM_used_by_node": 687 Mbs
"Node_info":
"BLS_key": 4N8aUNHSgjQVgkpm8nhNEfDf6txHznoYREg9kirmJrkivgL4oSEimFF6nsQ6M41QvhM2Z33nves5vfSn9n1UwNFJBYtWVnHYMATn76vLuL3zU88KyeAYcHfsih3He6UHcXDxcaecHVz6jhCYz1P2UZn2bDVruL5wXpehgBfBaLKm3Ba
"Catchup_status":
"Last_txn_3PC_keys":
"0":
"Node2":
None
None
"Node3":
None
None
"1":
"Node2":
None
None
"Node3":
None
None
"2":
"Node2":
None
None
"Node3":
None
None
"3":
"Node2":
None
None
"Node3":
None
None
"Ledger_statuses":
"0": synced
"1": synced
"2": synced
"3": synced
"Number_txns_in_catchup":
"0": 0
"1": 0
"2": 0
"3": 0
"Received_LedgerStatus":
"Waiting_consistency_proof_msgs":
"0": None
"1": None
"2": None
"3": None
"Client_ip": 0.0.0.0
"Client_port": 9702
"Client_protocol": tcp
"Committed_ledger_root_hashes":
"0": b'2PZguAeNfTpfPsavVTfAq98UC89Ko4DgbBa2CpTL1Svv'
"1": b'HgxLc6h6R12P5zSM1h7T5jSttmAoHryhmrshffmM9imx'
"2": b'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'
"3": b'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'
"Committed_state_root_hashes":
"0": b'EQUSi2crBb6G82h8NdUjU8MBNQaHTxRtwvtu9Qi1PAG'
"1": b'39aVhSB3idDKLkKTUGfKHFMhE2oNZ5GyCSDqCg4M5x2U'
"2": b'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA'
"Count_of_replicas": 2
"Freshness_status":
"0":
"Has_write_consensus": True
"Last_updated_time": 2024-07-07 23:30:43+00:00
"1":
"Has_write_consensus": True
"Last_updated_time": 2024-07-07 23:30:43+00:00
"2":
"Has_write_consensus": True
"Last_updated_time": 2024-07-07 23:30:43+00:00
"Metrics":
"Delta": 0.1
"Lambda": 240
"Omega": 20
"average-per-second":
"read-transactions": 0.0
"write-transactions": 0.0
"avg backup throughput": 0.0
"client avg request latencies":
"0": None
"1": None
"instances started":
"0": 1785.408685558
"1": 1785.410516859
"master throughput": None
"master throughput ratio": None
"max master request latencies": 0
"ordered request counts":
"0": 0
"1": 0
"ordered request durations":
"0": 0
"1": 0
"throughput":
"0": 0.0
"1": 0.0
"total requests": 0
"transaction-count":
"audit": 0
"config": 0
"ledger": 10
"pool": 4
"uptime": 300
"Mode": participating
"Name": Node1
"Node_ip": 0.0.0.0
"Node_port": 9701
"Node_protocol": tcp
"Replicas_status":
"Node1:0":
"Last_ordered_3PC":
0
0
"Primary": Node1:0
"Stashed_txns":
"Stashed_PrePrepare": 0
"Stashed_checkpoints": 0
"Watermarks": 0:300
"Node1:1":
"Last_ordered_3PC":
0
0
"Primary": Node2:1
"Stashed_txns":
"Stashed_PrePrepare": 0
"Stashed_checkpoints": 0
"Watermarks": 0:9223372036854775807
"Requests_timeouts":
"Ordering_phase_req_timeouts": 0
"Propagates_phase_req_timeouts": 0
"Uncommitted_ledger_root_hashes":
"Uncommitted_ledger_txns":
"0":
"Count": 0
"1":
"Count": 0
"2":
"Count": 0
"3":
"Count": 0
"Uncommitted_state_root_hashes":
"0": b'EQUSi2crBb6G82h8NdUjU8MBNQaHTxRtwvtu9Qi1PAG'
"1": b'39aVhSB3idDKLkKTUGfKHFMhE2oNZ5GyCSDqCg4M5x2U'
"2": b'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA'
"View_change_status":
"IC_queue":
"1":
"Voters":
"Node2":
"reason": 26
"Node3":
"reason": 26
"Last_complete_view_no": 0
"Last_view_change_started_at": 1970-01-01 00:00:00
"VCDone_queue":
"VC_in_progress": False
"View_No": 0
"did": Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv
"verkey": 33nHHYKnqmtGAVfZZGoP8hpeExeH45Fo8cKmd5mcnKYk7XgWNBxkkKJ
"Pool_info":
"Blacklisted_nodes":
"Quorums": {'n': 4, 'f': 1, 'weak': Quorum(2), 'strong': Quorum(3), 'propagate': Quorum(2), 'prepare': Quorum(2), 'commit': Quorum(3), 'reply': Quorum(2), 'view_change': Quorum(3), 'election': Quorum(3), 'view_change_ack': Quorum(2), 'view_change_done': Quorum(3), 'same_consistency_proof': Quorum(2), 'consistency_proof': Quorum(2), 'ledger_status': Quorum(2), 'ledger_status_last_3PC': Quorum(2), 'checkpoint': Quorum(2), 'timestamp': Quorum(2), 'bls_signatures': Quorum(3), 'observer_data': Quorum(2), 'backup_instance_faulty': Quorum(2)}
"Reachable_nodes":
['Node1', 0]
['Node2', 1]
['Node3', None]
['Node4', None]
"Reachable_nodes_count": 4
"Read_only": False
"Suspicious_nodes":
"Total_nodes_count": 4
"Unreachable_nodes":
"Unreachable_nodes_count": 0
"f_value": 1
"Protocol":
"Software":
"Indy_packages":
hi indy-node 1.13.2rc6 amd64 Indy node
hi indy-plenum 1.13.1rc4 amd64 Plenum Byzantine Fault Tolerant Protocol

 "Installed_packages":
       ioflo 2.0.2
       tzdata 2024.1
       distro 1.7.0
       indy-plenum 1.13.1rc4
       pillow 10.3.0
       rlp 0.6.0
       fonttools 4.53.0
       ujson 1.33
       python-rocksdb 0.7.0
       importlib_resources 6.4.0
       jsonpickle 3.0.1
       kiwisolver 1.4.5
       importlib-metadata 3.10.1
       zipp 3.19.2
       cycler 0.12.1
       python-ursa 0.1.1
       pyzmq 22.3.0
       leveldb 0.201
       matplotlib 3.7.5
       Pympler 0.8
       python-dateutil 2.8.2
       setuptools 50.3.2
       msgpack-python 0.5.6
       packaging 21.3
       timeout-decorator 0.5.0
       psutil 5.9.4
       pandas 2.0.3
       semver 2.13.0
       base58 2.1.1
       indy-node 1.13.2rc6
       orderedset 2.0.3
       sha3 0.2.1
       portalocker 2.7.0
       libnacl 1.6.1
       sortedcontainers 1.5.7
       contourpy 1.1.1
       prompt-toolkit 3.0.18
       numpy 1.24.4
       pytz 2024.1
       urllib3 1.25.8
       unattended-upgrades 0.1
       language-selector 0.1
       pyrsistent 0.15.5
       ubuntu-pro-client 8001
       SecretStorage 2.3.1
       pyOpenSSL 19.0.0
       zipp 1.0.0
       netifaces 0.10.4
       systemd-python 234
       PyJWT 1.7.1
       Automat 0.8.0
       requests 2.22.0
       sos 4.5.6
       PyYAML 5.3.1
       keyring 18.0.1
       more-itertools 4.2.0
       command-not-found 0.3
       hibagent 1.0.1
       jsonpointer 2.0
       pyasn1 0.4.2
       pluggy 0.13.0
       cryptography 2.8
       ec2-hibinit-agent 1.0.0
       idna 2.8
       PyHamcrest 1.9.0
       Twisted 18.9.0
       six 1.14.0
       python-apt 2.0.1+ubuntu0.20.4.1
       MarkupSafe 1.1.0
       python-debian 0.1.36+ubuntu1.1
       wadllib 1.3.3
       blinker 1.4
       hyperlink 19.0.0
       zope.interface 4.7.1
       pyasn1-modules 0.2.1
       PyNaCl 1.3.0
       pexpect 4.6.0
       Click 7.0
       httplib2 0.14.0
       certifi 2019.11.28
       jsonpatch 1.22
       pyparsing 2.4.6
       pyserial 3.4
       dbus-python 1.2.16
       service-identity 18.1.0
       entrypoints 0.3
       colorama 0.4.3
       oauthlib 3.1.0
       chardet 3.0.4
       PyGObject 3.36.0
       cloud-init 24.1.3
       distro-info 0.23+ubuntu1.1
       lazr.uri 1.0.3
       constantly 15.1.0
       attrs 19.3.0
       pip 20.0.2
       setuptools 45.2.0
       ufw 0.36
       wheel 0.34.2
       jsonschema 3.2.0
       Jinja2 2.10.1
       ssh-import-id 5.10
       requests-unixsocket 0.2.0
       launchpadlib 1.10.13
       incremental 16.10.1
       simplejson 3.16.0
       pymacaroons 0.13.0
       pytest 4.6.9
       atomicwrites 1.1.5
       wcwidth 0.1.8
       lazr.restfulclient 0.14.2
       configobj 5.0.6
       py 1.8.1
 "OS_version":   Linux-5.15.0-1064-aws-x86_64-with-glibc2.29
 "indy-node":    1.13.2rc6
 "sovrin":       unknown

"Update_time": Sunday, July 7, 2024 11:35:43 PM +0000
"response-version": 0.0.1
"timestamp": 1720395343

"Configuration":
"Config":
"Main_config":
# Current network

            # Disable stdout logging
            enableStdOutLogging = True
            #logLevel=0
            # Directory to store ledger.
            LEDGER_DIR = '/var/lib/indy'

            # Directory to store logs.
            LOG_DIR = '/var/log/indy'

            # Directory to store keys.
            KEYS_DIR = '/var/lib/indy'

            # Directory to store genesis transactions files.
            GENESIS_DIR = '/var/lib/indy'

            # Directory to store backups.
            BACKUP_DIR = '/var/lib/indy/backup'

            # Directory to store plugins.
            PLUGINS_DIR = '/var/lib/indy/plugins'

            # Directory to store node info.
            NODE_INFO_DIR = '/var/lib/indy'
            NETWORK_NAME = 'sandbox'
      "Network_config":
      "User_config":
 "Genesis_txns":
      "domain_txns":
            {"reqSignature":{},"txn":{"data":{"dest":"V4SGRU86Z58d6TV7PBUe6f","role":"0","verkey":"~CoRER63DVYnWZtK8uAzNbx"},"metadata":{},"type":"1"},"txnMetadata":{"seqNo":1},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"Th7MpTaRZVRYnPiabds81Y","role":"2","verkey":"~7TYfekw4GUagBnBVCqPjiC"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":2},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"EbP4aYNeTHL6q385GuVpRV","role":"2","verkey":"~RHGNtfvkgPEUQzQNtNxLNu"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":3},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"4cU41vWW82ArfxJxHkzXPG","role":"2","verkey":"~EMoPA6HrpiExVihsVfxD3H"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":4},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"TWwCRQRZ2ZHMJFn9TzLp7W","role":"2","verkey":"~UhP7K35SAXbix1kCQV4Upx"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":5},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"7JhapNNMLnwkbiC2ZmPZSE","verkey":"~LgpYPrzkB6awcHMTPZ9TVn"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":6},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"MEPecrczs4Wh6FA12u519D","verkey":"~A4rMgHYboWYS1DXibCgo9W"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":7},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"EAPtwgevBpzP8hkj9sxuzy","verkey":"~gmzSzu3feXC6g2djF7ar4"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":8},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"LuL1HK1sDruwkfm68jrVfD","verkey":"~Nyv9BKUJuvjgMbfbwk8CFD"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":9},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"462p8mtcX6jpa9ky565YEL","verkey":"~LCgq4hnSvMvB8nKd9vgsTD"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":10},"ver":"1"}

      "pool_txns":
            {"reqSignature":{},"txn":{"data":{"data":{"alias":"Node1","blskey":"4N8aUNHSgjQVgkpm8nhNEfDf6txHznoYREg9kirmJrkivgL4oSEimFF6nsQ6M41QvhM2Z33nves5vfSn9n1UwNFJBYtWVnHYMATn76vLuL3zU88KyeAYcHfsih3He6UHcXDxcaecHVz6jhCYz1P2UZn2bDVruL5wXpehgBfBaLKm3Ba","blskey_pop":"RahHYiCvoNCtPTrVtP7nMC5eTYrsUA8WjXbdhNc8debh1agE9bGiJxWBXYNFbnJXoXhWFMvyqhqhRoq737YQemH5ik9oL7R4NTTCz2LEZhkgLJzB3QRQqJyBNyv7acbdHrAT8nQ9UkLbaVL9NBpnWXBTw4LEMePaSHEw66RzPNdAX1","client_ip":"172.31.40.65","client_port":9702,"node_ip":"172.31.40.65","node_port":9701,"services":["VALIDATOR"]},"dest":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv"},"metadata":{"from":"Th7MpTaRZVRYnPiabds81Y"},"type":"0"},"txnMetadata":{"seqNo":1,"txnId":"fea82e10e894419fe2bea7d96296a6d46f50f93f9eeda954ec461b2ed2950b62"},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"data":{"alias":"Node2","blskey":"37rAPpXVoxzKhz7d9gkUe52XuXryuLXoM6P6LbWDB7LSbG62Lsb33sfG7zqS8TK1MXwuCHj1FKNzVpsnafmqLG1vXN88rt38mNFs9TENzm4QHdBzsvCuoBnPH7rpYYDo9DZNJePaDvRvqJKByCabubJz3XXKbEeshzpz4Ma5QYpJqjk","blskey_pop":"Qr658mWZ2YC8JXGXwMDQTzuZCWF7NK9EwxphGmcBvCh6ybUuLxbG65nsX4JvD4SPNtkJ2w9ug1yLTj6fgmuDg41TgECXjLCij3RMsV8CwewBVgVN67wsA45DFWvqvLtu4rjNnE9JbdFTc1Z4WCPA3Xan44K1HoHAq9EVeaRYs8zoF5","client_ip":"172.31.40.114","client_port":9704,"node_ip":"172.31.40.114","node_port":9703,"services":["VALIDATOR"]},"dest":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb"},"metadata":{"from":"EbP4aYNeTHL6q385GuVpRV"},"type":"0"},"txnMetadata":{"seqNo":2,"txnId":"1ac8aece2a18ced660fef8694b61aac3af08ba875ce3026a160acbc3a3af35fc"},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"data":{"alias":"Node3","blskey":"3WFpdbg7C5cnLYZwFZevJqhubkFALBfCBBok15GdrKMUhUjGsk3jV6QKj6MZgEubF7oqCafxNdkm7eswgA4sdKTRc82tLGzZBd6vNqU8dupzup6uYUf32KTHTPQbuUM8Yk4QFXjEf2Usu2TJcNkdgpyeUSX42u5LqdDDpNSWUK5deC5","blskey_pop":"QwDeb2CkNSx6r8QC8vGQK3GRv7Yndn84TGNijX8YXHPiagXajyfTjoR87rXUu4G4QLk2cF8NNyqWiYMus1623dELWwx57rLCFqGh7N4ZRbGDRP4fnVcaKg1BcUxQ866Ven4gw8y4N56S5HzxXNBZtLYmhGHvDtk6PFkFwCvxYrNYjh","client_ip":"172.31.32.116","client_port":9706,"node_ip":"172.31.32.116","node_port":9705,"services":["VALIDATOR"]},"dest":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya"},"metadata":{"from":"4cU41vWW82ArfxJxHkzXPG"},"type":"0"},"txnMetadata":{"seqNo":3,"txnId":"7e9f355dffa78ed24668f0e0e369fd8c224076571c51e2ea8be5f26479edebe4"},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"data":{"alias":"Node4","blskey":"2zN3bHM1m4rLz54MJHYSwvqzPchYp8jkHswveCLAEJVcX6Mm1wHQD1SkPYMzUDTZvWvhuE6VNAkK3KxVeEmsanSmvjVkReDeBEMxeDaayjcZjFGPydyey1qxBHmTvAnBKoPydvuTAqx5f7YNNRAdeLmUi99gERUU7TD8KfAa6MpQ9bw","blskey_pop":"RPLagxaR5xdimFzwmzYnz4ZhWtYQEj8iR5ZU53T2gitPCyCHQneUn2Huc4oeLd2B2HzkGnjAff4hWTJT6C7qHYB1Mv2wU5iHHGFWkhnTX9WsEAbunJCV2qcaXScKj4tTfvdDKfLiVuU2av6hbsMztirRze7LvYBkRHV3tGwyCptsrP","client_ip":"172.31.35.161","client_port":9708,"node_ip":"172.31.35.161","node_port":9707,"services":["VALIDATOR"]},"dest":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA"},"metadata":{"from":"TWwCRQRZ2ZHMJFn9TzLp7W"},"type":"0"},"txnMetadata":{"seqNo":4,"txnId":"aa5e817d7cc626170eca175822029339a444eb0ee8f0bd20d3b0b76e566fb008"},"ver":"1"}

 "indy-node-control.service":
       [Unit]
       Description=Service for upgrade of existing Indy Node and other operations
       #Requires=indy.service
       #After=indy.service
       After=network.target

       [Service]
       Type=simple
       EnvironmentFile=/etc/indy/node_control.conf
       ExecStart=/usr/bin/env python3 -O /usr/local/bin/start_node_control_tool $TEST_MODE --hold-ext ${HOLD_EXT}
       Restart=on-failure
       RestartSec=10
       StartLimitBurst=10
       StartLimitInterval=200
       TimeoutSec=300

       [Install]
       WantedBy=multi-user.target

 "indy-node.service":
       [Unit]
       Description=Indy Node
       Requires=indy-node-control.service

       [Service]
       EnvironmentFile=/etc/indy/indy.env
       ExecStart=/usr/bin/env python3 -O /usr/local/bin/start_indy_node ${NODE_NAME} ${NODE_IP} ${NODE_PORT} ${NODE_CLIENT_IP} ${NODE_CLIENT_PORT}
       User=indy
       Group=indy
       Restart=on-failure
       RestartSec=10
       StartLimitBurst=10
       StartLimitInterval=200
       TimeoutSec=300
       LimitNOFILE=65536:131072

       [Install]
       WantedBy=multi-user.target

 "indy.env":
       NODE_NAME=Node1
       NODE_IP=0.0.0.0
       NODE_PORT=9701
       NODE_CLIENT_IP=0.0.0.0
       NODE_CLIENT_PORT=9702
       CLIENT_CONNECTIONS_LIMIT=500
 "iptables_config":
 "node_control.conf":
       # Uncomment this to run agent in test mode:
       #TEST_MODE=--test

       TEST_MODE=
       HOLD_EXT=

In indy_config

Current network

Disable stdout logging

enableStdOutLogging = True

Directory to store ledger.

LEDGER_DIR = '/var/lib/indy'

Directory to store logs.

LOG_DIR = '/var/log/indy'

Directory to store keys.

KEYS_DIR = '/var/lib/indy'

Directory to store genesis transactions files.

GENESIS_DIR = '/var/lib/indy'

Directory to store backups.

BACKUP_DIR = '/var/lib/indy/backup'

Directory to store plugins.

PLUGINS_DIR = '/var/lib/indy/plugins'

Directory to store node info.

NODE_INFO_DIR = '/var/lib/indy'

indy.env
NODE_NAME=Node1
NODE_IP=0.0.0.0
NODE_PORT=9701
NODE_CLIENT_IP=0.0.0.0
NODE_CLIENT_PORT=9702

node_control.conf

Uncomment this to run agent in test mode:

#TEST_MODE=--test

TEST_MODE=
HOLD_EXT=