Unable to get scheduled reports

[Prathameshhankare]

Describe the bug

I am unable to get scheduled report on email whereas I get the email if I send the email manually(From GUI).

Error in Service:

[root@XXXXX01 ~]# systemctl status icinga-reporting.service -l
● icinga-reporting.service - Icinga Reporting Scheduler
Loaded: loaded (/etc/systemd/system/icinga-reporting.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sun 2024-07-21 13:45:03 UTC; 10min ago
Process: 27428 ExecStart=/usr/bin/icingacli reporting schedule run (code=exited, status=255)
Main PID: 27428 (code=exited, status=255)

Jul 21 13:39:22 XXXXX01 systemd[1]: Started Icinga Reporting Scheduler.
Jul 21 13:45:02 XXXXX01 icingacli[27428]: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8192 bytes) in /usr/share/icingaweb2/modules/pdfexport/vendor/textalk/websocket/lib/Base.php on line 88
Jul 21 13:45:02 XXXXX01 icingacli[27428]: Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8192 bytes) in /usr/share/icingaweb2/modules/pdfexport/vendor/textalk/websocket/lib/Base.php on line 88
Jul 21 13:45:02 XXXXX01 systemd[1]: icinga-reporting.service: main process exited, code=exited, status=255/n/a
Jul 21 13:45:03 XXXXX01 systemd[1]: Unit icinga-reporting.service entered failed state.
Jul 21 13:45:03 XXXXX01 systemd[1]: icinga-reporting.service failed.

Mail Logs:

Jul 21 13:55:01 XXXXX01 postfix/pickup[25322]: 4B1E5453B3A: uid=0 from=
Jul 21 13:55:01 XXXXX01 postfix/cleanup[30928]: 4B1E5453B3A: message-id=[email protected]
Jul 21 13:55:01 XXXXX01 postfix/qmgr[1347]: 4B1E5453B3A: from=[email protected], size=768, nrcpt=1 (queue active)
Jul 21 13:55:01 XXXXX01 postfix/pickup[25322]: 4FD0A453B41: uid=0 from=
Jul 21 13:55:01 XXXXX01 postfix/cleanup[30928]: 4FD0A453B41: message-id=[email protected]
Jul 21 13:55:01 XXXXX01 postfix/qmgr[1347]: 4FD0A453B41: from=[email protected], size=771, nrcpt=1 (queue active)
Jul 21 13:55:01 XXXXX01 postfix/local[30942]: 4B1E5453B3A: to=[email protected], orig_to=, relay=local, delay=0.13, delays=0.09/0.02/0/0.02, dsn=2.0.0, status=sent (delivered to mailbox)
Jul 21 13:55:01 XXXXX01 postfix/qmgr[1347]: 4B1E5453B3A: removed
Jul 21 13:55:01 XXXXX01 postfix/local[30944]: 4FD0A453B41: to=[email protected], orig_to=, relay=local, delay=0.17, delays=0.09/0.04/0/0.03, dsn=2.0.0, status=sent (delivered to mailbox)
Jul 21 13:55:01 XXXXX01 postfix/qmgr[1347]: 4FD0A453B41: removed
Jul 21 14:00:01 XXXXX01 postfix/pickup[25322]: D3DF1453B40: uid=0 from=
Jul 21 14:00:01 XXXXX01 postfix/cleanup[31941]: D3DF1453B40: message-id=[email protected]
Jul 21 14:00:01 XXXXX01 postfix/qmgr[1347]: D3DF1453B40: from=[email protected], size=771, nrcpt=1 (queue active)
Jul 21 14:00:01 XXXXX01 postfix/pickup[25322]: D9CF7453B41: uid=0 from=
Jul 21 14:00:01 XXXXX01 postfix/cleanup[31941]: D9CF7453B41: message-id=[email protected]
Jul 21 14:00:01 XXXXX01 postfix/qmgr[1347]: D9CF7453B41: from=[email protected], size=768, nrcpt=1 (queue active)
Jul 21 14:00:01 XXXXX01 postfix/local[31954]: D3DF1453B40: to=[email protected], orig_to=, relay=local, delay=0.09, delays=0.07/0.02/0/0, dsn=2.0.0, status=sent (delivered to mailbox)
Jul 21 14:00:01 XXXXX01 postfix/qmgr[1347]: D3DF1453B40: removed
Jul 21 14:00:01 XXXXX01 postfix/local[31954]: D9CF7453B41: to=[email protected], orig_to=, relay=local, delay=0.09, delays=0.06/0.02/0/0.01, dsn=2.0.0, status=sent (delivered to mailbox)
Jul 21 14:00:01 XXXXX01 postfix/qmgr[1347]: D9CF7453B41: removed

Your Environment

• Module version: v1.0.1
• Dependent module versions: pdfexport v0.11.0
• Icinga Web 2 version and modules (System - About): 2.12.1
• Icinga 2 version used (icinga2 --version): r2.13.5-1
• PHP version used (php --version): 7.2.34
• Server operating system and version: CentOS 7

[gurhanocaktan]

I have the same issue

[sukhwinder33445]

Hi @Prathameshhankare @gurhanocaktan,
Can you please share the debug log (icingacli reporting schedule run --debug).
Do you have a cover page with image/logo? What size are these images?
How many objects are included in the report you want to send?

[gurhanocaktan]

Hi @sukhwinder33445

icingacli reporting schedule run --debug
This only returns me
Scheduling job Schedule1 to run at 2024-07-26 06:00:00

There are 242 services I have. I wanted to send SLA of all these objects.
I have a logo in cover page. The size of the image must be 350 x 154 .

When I click send and manually trigger , I receive the email and report .

[sukhwinder33445]

Hi @gurhanocaktan,

Yes, this schedule will be executed tomorrow.

You can create a new schedule for tests with a minutely frequency.
Then run the same command.

[gurhanocaktan]

Hi @sukhwinder33445 ,
When I made it minutely , I received the report succesfully 👀 .

[sukhwinder33445]

Hi @gurhanocaktan,

Maybe this was just a one-time error, this can happen if the data and the cover sheet are very large.
I am closing this issue for now. If it occurs again, please reopen the issue and share your debug log.

[gurhanocaktan]

Hi @sukhwinder33445 ,

I change the schedule again to daily . I will check if I receive or not tomorrow .
I do not think it was one-time error because I created new reports and make daily . I have never received them .
Maybe bug is related to daily setting . idk.

[gurhanocaktan]

Yes I did not get the daily one again. :/

[sukhwinder33445]

Can you please provide the debug log from today.
Please make sure you have the latest version 1.0.2 of this module and 0.11.1 (latest) of the pdfexport module.

[gurhanocaktan]

Hi @sukhwinder33445 ,

I am using the latest version of reporting module but ı realized I am using 0.11.0 module of pdfexport . I will upgrade it .
Though I do not think this is the problem . When I make the occurancy minutely I get the emails and report.

How can I see the debug log from today ?

[sukhwinder33445]

The logs of the reporting module should be included in your icingaweb2 logs.

[sukhwinder33445]

Hi @gurhanocaktan,

Do you still have the issue even after updating pdfexport and the reporting module to the latest version?

[sukhwinder33445]

Hi @gurhanocaktan,

If you cannot find the logs of the report module, please try the following:

1. Edit the file /etc/systemd/system/icinga-reporting.service.
   • Add the --debug flag : ExecStart=/usr/bin/icingacli reporting schedule run --debug
2. Reload the units: systemctl daemon-reload
3. Restart the service: systemctl restart icinga-reporting.service
4. Make sure that the service is running with the debug flag: systemctl status icinga-reporting.service
5. Now you should see the logs: journalctl -u icinga-reporting.service --no-pager

[gurhanocaktan]

Hi @sukhwinder33445 ,

I still have the issue even tho I have the latest version of these modules ( I already had the latest versions ) .

I scheduled the daily report 5 minutes later and sometimes it worked. However, the day after it did not send again .
I am thinking it might be about permission issues maybe .

I will try to get these logs and send you

[gurhanocaktan]

Btw I realized something odd ,

When I schedule the report for at 1.20pm ,
This is what debug returns :

icingacli reporting schedule run --debug
Scheduling job Schedule1 to run at 2024-08-01 11:20:00

There is 2 hours difference.

[Prathameshhankare]

@sukhwinder33445, please take a look at the below debug log you requested.
debuglog.txt

[Prathameshhankare]

@sukhwinder33445, please take a look at the below debug log you requested. debuglog.txt

Any update on the shared debug logs?

[sukhwinder33445]

Hi @Prathameshhankare,

Unfortunately, the shared log file does not contain a service crash log, only normal logs, which are harmless.

But I was able to reproduce it once. I am looking for the cause.

[Prathameshhankare]

Hi @sukhwinder33445,

Have we identified the cause? Please let me know if we have a solution for this issue.

[Prathameshhankare]

Hi @sukhwinder33445,

Could you please provide an update on the progress of identifying the root cause? I’m awaiting your response.

[sukhwinder33445]

Hi @Prathameshhankare,

I have successfully reproduced the crash of the daemon and acknowledged the bug. Unfortunately, I cannot give you a date when this bug will be fixed.

As a workaround, you can omit the template for reports, which leads to a lower memory usage. Please let me know if this works for you.

[Prathameshhankare]

@sukhwinder33445, I tried the workaround, omitted the template for reports, but still no luck.

[sukhwinder33445]

Hi @Prathameshhankare,

Is it a large file (how many objects)? How many schedules do you have, and when are they triggered (frequency, time)?
You can try to reduce the report size using the filter (still without template).