Avoid static list of Addresses in ConnectionHandler

[Srinathc]

Setup:

• an auto-scaling cluster of rabbitmq nodes

Problem:
The size of the rabbitmq cluster can scale up and down. But once a connection has been setup, the list of addresses cannot be updated without a restart or by discarding the connection and re-establishing the entire connection again.

Proposal:
Provide a dereferencing mechanism to obtain the list of Addresses by invoking a callback instead of maintaining a static list of Addresses.

Doing so solves the following use-cases:

• For long running (days, months, etc) AMQP clients there would be no restart required even if the entire rabbitmq cluster was re-created on different systems. The deference callback can load and return the latest list of addresses.
• Balance the load across rabbitmq nodes - the callback can use different mechanisms and find the least loaded server to connect to rather than the ConnectionHandler trying them in sequence.
• Integrate with discovery/directory services like ldap, zookeeper or netflix-eureka - each time a connection recovery is attempted, the latest state of the rabbitmq cluster can be read and returned

Let me know your thoughts and I am willing to contribute in any way on this.

[jhalterman]

So basically you just want to change a Connection's options after it's been created?

Right now the ConnectionOptions class is cloned when a Connection is created, and the new clone is stored internally for the Connection. If addresses are something that can change over time (like configuration) I'd consider either moving them to the ConnectionConfig interface where they can be exposed and modified over time or making the ConnectionOptions publicly available via the ConfigurableConnection interface.

Conceptually, options were intended to be things that do not change whereas configuration can change. You've brought up a good point that for connections which can re-established, the config could potentially change.

Do you have any API suggestions for what this would look like?

[Srinathc]

Thanks for explaining the design considerations of ConnectionOptions and ConnectionConfig. I now understand the separation of concerns.

Proposals for address resolver
Option 1 - A very simple one, and should suffice for most cases and good to start with

   // A resolver that is called back when a new (or recovery) connection is attempted       
   public interface AddressResolver {
        Address[] resolve();
   }

Option 2 - A stateful resolver - helps to pick an address that has lesser number of open connections, or is located in the same zone/region. The entire logic can be plugged in this way.

   // A resolver that is called back when a new (or recovery) connection is attempted
   public interface AddressResolver {
        Address nextAddress();
   }

Option 2 would be my preference since it caters to more usecases.

Proposals for setting the address resolver
I would be ok to have the resolver set into either the Options or the Config.

[Srinathc]

Please review and comment on the proposed changes as in the above comment.

[Srinathc]

Have raised a pull request for this with Option 2. See #21.
Happy to incorporate any review comments suggested.

[jhalterman]

Still thinking about this feature. You described a few possible use cases for an AddressResolver, but I'm wondering how you'll most likely use this in practice? I'm just wondering about the necessity of having something like an AddressResolver versus simply being able to change the set of addresses (options.setAddresses) after a Connection has been established. Along the same lines, it makes me wonder if there's a need to change other ConnectionOptions after a Connection has been established - probably not, but if so I'd like to address all of this at the same time.

[Srinathc]

In an environment that I'm working on right now, I would implement the AddressResolver to fetch the current list of addresses from a central registry. This is a http call to an external system which maintains the rabbitmq cluster member details and it would take a few milli-seconds to complete.

Having to set "options.setAddresses" is ok, but the point is that I would need to update the addresses periodically anticipating a connection re-establishment. Having a resolver helps to optimize in saying that the fetch is going to be made only when required (i.e. when a new connection is being re-attempted).

I haven't come across use-cases for updating the connection factory settings or the execution service after connection establishment. But if I do, I'll raise them here.