KS service shutdown is stalling writing checkpoint file

[big-andy-coates]

As it stands, when the system test shuts down a KS service, the service doesn't complete shutting down cleanly.

The ProcessorStateManager class in streams is calling:

checkpointFile.write(checkpointingOffsets);

Which kind of blocks... ish... but not. There aren't any checkpoints to write, and the state.dir doesn't exist. Yet the call blocks for 10+ seconds. After which, the JVM shuts down.

The System test executor has a 30s timeout for graceful shutdown and this is not hit.
The streams app defaults to a 30s timeout for graceful shutdown, and this isn't hit either.

So actually not sure what is causing it to shutdown.

However, as we haven't yet written the code for adding mount points for state to KS apps... maybe its something to do with that.