OutOfMemoryError since upgrading to version 0.5.0

[JorgeAlmeida]

We had some issues with OutOfMemoryErrors since upgrading to version 0.5.0.

We were running version 0.4.1 for the last 6 months without issues and upgraded to 0.5.0 with the same volume of events, same configuration thresholds and same machine resources.
After this upgrade we registered OOM errors and the application did not recover from them. We restarted the containers a few times but always ending up with the same result.
We fixed this by moving to larger instances types, but with the same volume of events as before this was not expected.

More info:
sinkLzoBufferByteThreshold=500000000
sinkLzoBufferRecordThreshold=2000000
sinkLzoBufferTimeThreshold=3600000

[BenFradet]

@JorgeAlmeida There are two things to take into consideration:

• Before 0.5.0, OOMs were swallowed by the KCL and now they're bubbling so it's most likely that you had those before but they weren't crashing the application.
  If you still have the logs for 0.4.1 I invite you to look for "Error writing raw stream to output stream" log messages which were indicating ooms.
• amazon-kinesis-connectors, the library we're using to read from Kinesis, has no backpressure which means that if, for some reason, we accumulate delay sending events to s3, the buffer will keep on growing until an oom is reached.