Cannot establish two fix connections to two different clients in same classloader

[kotari4u]

This is continuation for bug #428. It has been suggested to post the query here. I can post the latest response here to continue the discussion

#428

Thank you and appreciate for the quick response.
I cannot have one ThreadedSocketInitiator and have N number of fix sessions, due to few design constraints. I need two separate ThreadedSoketInitiator and when I initiate two beans in different/same spring context for two different clients, then I see problem at the below code, where we are scheduling a fixed task.

Lets say one ThreadedSocketInitiator comes and schedule a fixed task for Vendor 1 connectivity and when we create another bean it will overwrite the scheduled task because "SCHEDULED_EXECUTOR" is private "static" final variable. As it is static only one executor service is allowed and only one task can be run. If it non-static private final then we will have two executor services and its own scheduled jobs. Any reason it has to be "static". Removing "static" here will solve this issue. Not sure if anyone else faced this problem. But if we create two ThreadedSocketInitiator's and connecting to two different clients from the same class loader then we would see this issue

protected void startSessionTimer() {
Runnable timerTask = new SessionTimerTask();
if (shortLivedExecutor != null) {
timerTask = new DelegatingTask(timerTask, shortLivedExecutor);
}
sessionTimerFuture = SCHEDULED_EXECUTOR.scheduleAtFixedRate(timerTask, 0, 1000L,
TimeUnit.MILLISECONDS);
log.info("SessionTimer started");
}

[chrjohn]

On behalf of @esanchezros (copied from #428 (comment)):

Hi @kotari4u,

I don't quite understand your problem. The ThreadedSocketInitiator is not used for scheduling tasks. The Initiator accepts an ExecutorFactory that can be used to configure the Executors for its internal use only. If you need to schedule a task you can use a different ExecutorService.

Can you create a sample project illustrating the issue you're having?

Thanks
Ed

[kotari4u]

Thank you for the response. I tried to create a sample project but could not replicate issue. We will dig further and get back if we can replicate. Thank you..

[chrjohn]

Thanks for the feedback.

[QwertGold]

I spent a little time digging into this, as I also have a Spring application with beans wrapping multiple SocketInitiator.

Lets look at

private static final ScheduledExecutorService SCHEDULED_EXECUTOR = Executors
            .newSingleThreadScheduledExecutor(new QFTimerThreadFactory())

Here static just means the ScheduledExecutorService is constructed when the class is loaded.

This ScheduledExecutorService is only used in one location, which looks like this

Runnable timerTask = new SessionTimerTask();
if (shortLivedExecutor != null) {
    timerTask = new DelegatingTask(timerTask, shortLivedExecutor);
}
sessionTimerFuture = SCHEDULED_EXECUTOR.scheduleAtFixedRate(timerTask, 0, 1000L,
        TimeUnit.MILLISECONDS);

This means that if the non-static shortLivedExecutor is defined, the actual task is delegated to that Executor, and no work happens on the ScheduledExecutorService.

The shortLivedExecutor is only set if setExecutorFactory is called. QuickFixJ does not do this automatically. This means that all timer tasks (classloader scope) will be run sequentially, unless you call SocketInitiator.setExecutorFactory to configure an Executor with the number of threads you would like.

A smal warning: When reading the code, the intention of setExecutorFactory is quite clear, but there are 0 calls to this method, which probably means the functionality is not covered by test. I also think it is unlikely that many other QuickFixJ users have called this method, so you may be venturing into uncharted territory; I have used QuickFixJ since 2014 and only discovered this behavior today.

Why static threads are bad
I have the pleasure of working with QuickFixJ inside a Web Container (Tomcat). There are two characteristics of web container that should be considered with relation to threads.

1. Each application has its own class loader
2. The lifetime of the container can be longer than the application

In the old days our Tomcat would run for weeks, but we would deploy our application continuously, and every time we did we would get this warning

The web application [XYZ] appears to have started a thread named [QFJ Timer] but has failed to stop it. This is very likely to create a memory leak.

In normal applications you mark you thread as a daemon, so it does not prevent the JVM from shutting down. This works because the JVM and application have the same lifecycle. When you have a container based system with multiple class loaders, unstopped threads will prevent GC of anything that is reachable by that thread.
These days most new java systems are deployed inside Docker containers, an although many run inside a web container the JVM and application have the same lifecycle, making static threads a non issue for most devOps today.
I still try to teach the importance of thread lifecycle to new developers, as the principle translates into things like Spring's Lifecycle

[kotari4u]

Thanks much for the detailed response. We tried to overwrite this scheduled service but if I remember correctly, there is no setter method to set this. I will double check once.

[QwertGold]

There is no way to configure the ScheduledExecutorService , instead you should configure the Executor it delegates to, by calling setExecutorFactory

SocketInitiator initiator = new SocketInitiator(clientApplication, new MemoryStoreFactory(),
        settings, new DefaultMessageFactory());

initiator.setExecutorFactory(new ExecutorFactory() {
    @Override
    public Executor getLongLivedExecutor() {
        return Executors.newSingleThreadExecutor();
    }

    @Override
    public Executor getShortLivedExecutor() {
        return Executors.newSingleThreadExecutor();
    }
});