Improve re connection to data endpoints based on configurable values.

[shilmyhasan]

Purpose

resolves wso2/api-manager#407

Currently the re connection to a failed data endpoints run as a scheduled service with a configurable fixed time gap. This creates lot of logs when a data endpoint goes down in regular interval and create performance impacts while trying to re connect to a endpoint on same fixed intervals. This behavior has been improved to increase the delay between each failed connection attempt exponentially through configurable value to reduce the re connection overhead of a failed endpoint.

Implementation

This is implemented with three configurable properties namely ReConnectionInterval, ExpFacto, MaxReConnectionInterval. The delay between each failed attempt will be increased by ReConnectionInterval * ExpFactor till it reaches the MaxReconnectionInterval. Once it reaches the MaxReconnectionInterval , then every other failed re connection attempts will be executed after the MaxReconnectionInterval. If a connection attempt is succesfull when the data endpoint is running again, then the re connection interval will be re set to its old configured value.

Sample Configurations

[transport.binary.agent]
reconnection_interval = 30
exp_factor=2
max_reconnection_interval = 3600

Sample Behavior with the above configurations

Request attempts

1st failed connection attempt - Adds delay of (30*2) 60 seconds for the next attempt from the current time.

2nd failed connection attempt - Adds delay of (60*2) 120 seconds for the next attempt from the current time.

3rd failed connection attempt - Adds delay of (120*2) 240 secods for the next attempt from the current time.

4th succesffull connection attempt - Resets the delay to its old value and no connection attempt will be made unless the endpoints state changes to UNAVAIABLE.

Note : If someone needs to keep the old behavior to keep retrying in the fixed intervals , then exp_factor factor should be set to 1.