Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Count the number of throttled log message and print it. #1520

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

ovalenti
Copy link
Contributor

@ovalenti ovalenti commented Jan 26, 2024

Description

We have recently increased significantly the delay for some throttled messages.

In order to get a better idea of the amount of messages, this PR adds a counter.

Checklist

inspected CI test results for throttled messages:

  • Seen a message not throttled.
  • Seen a message which was throttled.

Testing

With a container generating 100 zombie processes, the collector contains:

...
[INFO    2024/07/05 19:12:09] Successfully established GRPC stream for signals.
[INFO    2024/07/05 19:12:09] Found self-check process event.
[ERROR   2024/07/05 19:12:09] Could not determine network namespace: No such file or directory
[INFO    2024/07/05 19:12:10] Found self-check connection event.
[INFO    2024/07/05 19:12:19] self-check (pid=76) exited with status: 0
[INFO    2024/07/05 19:12:39] Flushing thread table
[INFO    2024/07/05 19:12:39] Flushing container table
[ERROR   2024/07/05 19:12:39] [Throttled 99 messages] Could not determine network namespace: No such file or directory
[INFO    2024/07/05 19:13:09] Flushing container table
[ERROR   2024/07/05 19:13:09] [Throttled 99 messages] Could not determine network namespace: No such file or directory
...

@ovalenti ovalenti force-pushed the ovalenti/count-throttled-logs branch from cc4a658 to f3e5456 Compare July 4, 2024 15:54
@ovalenti ovalenti force-pushed the ovalenti/count-throttled-logs branch from 925e2eb to f68248b Compare July 4, 2024 16:24
This is likely less efficient, but prevents warnings about ambiguous
dangling else statement.
@ovalenti ovalenti force-pushed the ovalenti/count-throttled-logs branch from 2b8fe96 to 5e68bb8 Compare July 5, 2024 18:22
@ovalenti ovalenti force-pushed the ovalenti/count-throttled-logs branch from 5e68bb8 to fb917d9 Compare July 5, 2024 19:01
@ovalenti ovalenti marked this pull request as ready for review July 5, 2024 19:57
@ovalenti ovalenti requested a review from a team as a code owner July 5, 2024 19:57
static std::chrono::steady_clock::time_point _clog_lastlog_##__LINE__; \
static unsigned long _clog_throttle_times_##__LINE__ = 0; \
std::chrono::duration _clog_elapsed_##__LINE__ = std::chrono::steady_clock::now() - _clog_lastlog_##__LINE__; \
if (collector::logging::CheckLogLevel(collector::logging::LogLevel::lvl) && (cond) && _clog_elapsed_##__LINE__ < interval) { \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not been able to refactor this to use nested ifs. I think this is okay.

@JoukoVirtanen JoukoVirtanen self-requested a review August 1, 2024 21:03
@JoukoVirtanen
Copy link
Contributor

I will approve after merge conflicts are fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants