StackOverflow when running large reports

[Murilocb123]

Hello,
When executing large reports, which have many records, around 10 thousand pages, yes I have clients who issue this exaggerated number of pages, but for certain reasons I am forced to allow them to issue this amount of records, however when updating the version 4.4 to 4.10 of birt I encounter a StackOverflow error, but in the previous version it did not happen. I needed to know if it was known to you or if it was resolved in future versions.

• Error:

15-Sep-2023 08:16:00.953 ADVERTÊNCIA [http-nio-8080-exec-5] br.com.ipm.birt.report.util.ReportLogger.log Erro durante o processamento do relatorio.
org.eclipse.birt.report.engine.api.EngineException: Error happened while running the report.
at org.eclipse.birt.report.engine.api.impl.EngineTask.handleFatalExceptions(EngineTask.java:1877)
at org.eclipse.birt.report.engine.api.impl.RunAndRenderTask.doRun(RunAndRenderTask.java:153)
at org.eclipse.birt.report.engine.api.impl.RunAndRenderTask.run(RunAndRenderTask.java:69)
at br.com.ipm.report.servlet.ServletBirt.sendReportDirect(ServletBirt.java:176)
at br.com.ipm.report.servlet.WebReport.doPost(WebReport.java:133)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:696)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.StackOverflowError
at java.base/java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1541)
at java.base/java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:666)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1252)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:574)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:574)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
at org.eclipse.birt.report.engine.executor.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:95)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:574)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:574)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:574)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:574)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:574)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:574)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:574)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:574)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:574)
...

Environment:

• Docker
• Container with a maximum of 6 GB
• HeapSpace limited to 5gb
• MetaSpace at 256M

Observation:

• I noticed that when I don't limit MetaSpace the error doesn't happen
• Could it be related to the environment or some bug?

[speckyspooky]

Wow, I read that you have more like tenthousand pages with your limitation of the HeapSpace.
So I work a lot of reports with images and for example my local HeapSpace of the eclipse-designer (all-in-one) is set to 10GB
because on large reports (here I mean with lot of images) I get limits to but I have a very lower page count like you.
My QS-System is higher sized so the virtual machine has 32GB and the JVM of BIRT got 24GB from my side.

Therefore it could be that some changes of newer BIRT-versions will need a little bit more RAM.
But in general, I'm wondering that your mega-reports was running with you scalled container.

[Murilocb123]

Sorry I got confused, it's around 5 thousand pages (it could be longer). But increasing resources is not an option, so I'm struggling to adjust these points, but I discovered that generally in reports without a cross tab they actually issue, that is, what is causing this stack overflow is in reports with a crosstab. As a temporary measure, I increased the size of the java -Xss stack, it solved it for now, but I would like to understand if it was something introduced in more recent versions of birt, as I previously used 4.4 and did not generate these problems, of course in a structure without docker, with a server for all our java applications. Sorry, I'm using the translator, I'm not good with English 😄.

[hvbtup]

Probably your report was right below some memory limit with BIRT 4.4, and now you hit the limit.
It is perfectly normal that a new release of a software needs more (or less) memory than before.
For example, a newer release may have more properties for each item than the old release.
But as long as the new release doesn't need a lot more memory, there's hardly any reason to investigate the reasons further.

Keep in mind that for a giant cross tab, BIRT needs to keep all the layout items in memory.
You should perhaps re-think the choice of your report design.

Note that if your data source is an SQL database, you could use pure SQL with grouping and analytic functions for the heavy lifting,
resulting in a set with columns like (assuming 3 columns fit on a single page)

matrix_number, -- If you crosstab needs e.g. 12 columns, this runs from 1 to 4 (e.g. the second matrix shows columns 4,5,6)
row_number,
row_title_a, row_title_b, -- what you want to display to the left side of the matrix
col_head_a_1,
col_head_a_2,
col_head_a_3,
col_head_b_1,
col_head_b_2,
col_head_b_3,
cell_value

You can then use a simple table item instead of a cross tab to display the results.
The table item needs a group based on matrix_number.
In the binding, you add an MAX aggregate for the col_head_a_1 column at the group level (similar for col_head_a_2, col_head_b_1, col_head_b_2).

Then in the layout you display these aggregates in the group header line.

I describe this in my answer at https://stackoverflow.com/questions/72831123/reporting-with-sub-tables

[Murilocb123]

I understand, that's what I imagined regarding loading into memory, thanks for the help @hvbtup and @speckyspooky.