Feat: Log dumper #445

waldgange · 2024-10-07T18:03:33Z

Introduce a set of parameters for log record dumping by a trigger message. This section can be added to the Broker config:

{
    ....
    "taskConfig": {
        ...
        "logDump": {
            "recordBufferSize": 32768,
            "recordingLevel": "TRACE",
            "triggerLevel": "FATAL"
        }
        ....
    }
    ....
}

Where:
recordBufferSize is a size of default logger's record buffer in bytes.
recordingLevel is the lowest severity level of the record that will be stored by the logger in its log record buffer.
triggerLevel is the lowest severity level of the record that will cause immediate publication of that record and any records in the logger's log record buffer.

bmq-oss-ci

Build 294 of commit 004c2fc has completed with FAILURE

bmq-oss-ci

Build 298 of commit 28ec79b has completed with FAILURE

bmq-oss-ci

Build 300 of commit 8ce4504 has completed with FAILURE

chrisbeard · 2024-10-09T23:43:18Z

Looks like there's a unit test for the log controller, let's extend that to test this as well.

I think the most important thing to make sure of with this test is that we don't break compatibility with older configs that don't utilize this.

waldgange · 2024-10-16T13:09:24Z

@chrisbeard I've added a unit test for LogControllerConfig::fromObj(), also I've removed the outdated method fromDatum() and its unit test.
In order not to break compatibility with old broker configs we have to release this feature in two steps. As the first step we release the new config, containing the new parameters. When all machines upgrade to this version we will release the broker that retrieves new parameters from the config

bmq-oss-ci

Build 310 of commit a63f4e7 has completed with FAILURE

bmq-oss-ci

Build 351 of commit 7b9f24e has completed with FAILURE

bmq-oss-ci

Build 358 of commit f645c26 has completed with FAILURE

bmq-oss-ci

Build 360 of commit c3ef08d has completed with FAILURE

bmq-oss-ci

Build 372 of commit 6a93d60 has completed with FAILURE

Signed-off-by: Anton Pryakhin <[email protected]>

bmq-oss-ci

Build 378 of commit 84595e7 has completed with FAILURE

hallfox · 2024-11-11T17:38:41Z

src/groups/bmq/bmqtsk/bmqtsk_logcontroller.cpp

@@ -147,6 +147,9 @@ LogControllerConfig::LogControllerConfig(bslma::Allocator* allocator)
 , d_syslogAppName("")
 , d_syslogVerbosity(ball::Severity::ERROR)
 , d_categories(allocator)
+, d_recordBufferSize(32768)


Where does this number come from?

This is the default recordBufferSize of the ball::LoggerManagerConfiguration component we use to dump logs. We used to set it here

blazingmq/src/groups/bmq/bmqtsk/bmqtsk_logcontroller.cpp

Line 654 in b390620

rc = lmc.setDefaultRecordBufferSizeIfValid(32768);

Now it's a variable. We are going to retrieve this number from the config

Is this number in bytes? Can we just quickly mark the unit for it?

Yes. I've renamed the variable and added a comment in the same style as the variables above.

hallfox · 2024-11-11T17:39:20Z

src/groups/bmq/bmqtsk/bmqtsk_logcontroller.h

+    // TODO: use obj.logDump() when the config is updated
+    // .setRecordBufferSize(obj.logDump().recordBufferSize());


Ok I will remove this. This comment is a reminder to make the second step when all instances get the new version of the config. We used to leave such reminders in the past. Should we avoid them now?

hallfox · 2024-11-11T17:39:46Z

src/groups/bmq/bmqtsk/bmqtsk_logcontroller.h

+    // if (ball::SeverityUtil::fromAsciiCaseless(
+    //         &d_recordingVerbosity,
+    //         obj.logDump().recordingLevel().c_str()) != 0) {
+    //     errorDescription << "Invalid value for 'recordingLevel' ('"
+    //                      << obj.logDump().recordingLevel() << "')";
+    //     return -1;  // RETURN
+    // }
+
+    // if (ball::SeverityUtil::fromAsciiCaseless(
+    //         &d_triggerVerbosity,
+    //         obj.logDump().triggerLevel().c_str()) != 0) {
+    //     errorDescription << "Invalid value for 'triggerLevel' ('"
+    //                      << obj.logDump().triggerLevel() << "')";
+    //     return -1;  // RETURN
+    // }


hallfox · 2024-11-11T17:44:54Z

src/groups/bmq/bmqtsk/bmqtsk_logcontroller.t.cpp

+    // TODO: uncomment when setup of this parameter is uncommented
+    // lc.bslsLogSeverityThreshold() = "error";


removed this

hallfox · 2024-11-11T17:46:14Z

src/groups/bmq/bmqtsk/bmqtsk_logcontroller.h

-    /// might be partially altered.  Refer to the component level
-    /// documentation (section: "LogControllerConfig: Datum format") for the
-    /// expected format of the `datum`.
-    int fromDatum(bsl::ostream& errorDescription, const bdld::Datum& datum);


Why is this being deleted?

We don't use it anywhere since 2019. We use 'fromObj()' method to parse the broker config instead

Signed-off-by: Anton Pryakhin <[email protected]>

src/groups/mqb/mqbcfg/mqbcfg_messages.cpp

hallfox · 2024-11-27T16:26:23Z

src/groups/bmq/bmqtsk/bmqtsk_logcontroller.cpp

@@ -147,6 +147,9 @@ LogControllerConfig::LogControllerConfig(bslma::Allocator* allocator)
 , d_syslogAppName("")
 , d_syslogVerbosity(ball::Severity::ERROR)
 , d_categories(allocator)
+, d_recordBufferSize(32768)


Is this number in bytes? Can we just quickly mark the unit for it?

Co-authored-by: Taylor Foxhall <[email protected]> Signed-off-by: Anton Pryakhin <[email protected]>

Signed-off-by: Anton Pryakhin <[email protected]>

bmq-oss-ci

Build 397 of commit 93c314c has completed with FAILURE

waldgange requested a review from a team as a code owner October 7, 2024 18:03

waldgange force-pushed the logdumper branch from 10fd3bd to 2585b47 Compare October 8, 2024 08:44

waldgange changed the title ~~Log dumper~~ Feat: Log dumper Oct 8, 2024

waldgange force-pushed the logdumper branch from 2585b47 to 004c2fc Compare October 8, 2024 08:47

bmq-oss-ci bot reviewed Oct 8, 2024

View reviewed changes

chrisbeard self-assigned this Oct 9, 2024

bmq-oss-ci bot reviewed Oct 9, 2024

View reviewed changes

waldgange force-pushed the logdumper branch 3 times, most recently from 9a30dd5 to b827d6f Compare October 16, 2024 08:35

bmq-oss-ci bot reviewed Oct 16, 2024

View reviewed changes

waldgange force-pushed the logdumper branch 4 times, most recently from 5a0a785 to 7b9f24e Compare November 1, 2024 10:09

bmq-oss-ci bot reviewed Nov 1, 2024

View reviewed changes

waldgange force-pushed the logdumper branch from 7b9f24e to f645c26 Compare November 6, 2024 14:37

bmq-oss-ci bot reviewed Nov 6, 2024

View reviewed changes

waldgange force-pushed the logdumper branch from f645c26 to c3ef08d Compare November 7, 2024 14:40

bmq-oss-ci bot reviewed Nov 7, 2024

View reviewed changes

waldgange force-pushed the logdumper branch from c3ef08d to 6a93d60 Compare November 14, 2024 08:49

bmq-oss-ci bot reviewed Nov 14, 2024

View reviewed changes

Feat[MQB]: Log dumper

84595e7

Signed-off-by: Anton Pryakhin <[email protected]>

waldgange force-pushed the logdumper branch from 6a93d60 to 84595e7 Compare November 18, 2024 11:40

bmq-oss-ci bot reviewed Nov 18, 2024

View reviewed changes

hallfox reviewed Nov 19, 2024

View reviewed changes

Cleanup comments

bdd36d1

Signed-off-by: Anton Pryakhin <[email protected]>

Merge branch 'main' into logdumper

8479e38

waldgange requested a review from hallfox November 22, 2024 09:56

hallfox reviewed Nov 27, 2024

View reviewed changes

waldgange and others added 3 commits December 5, 2024 13:31

Update src/groups/mqb/mqbcfg/mqbcfg_messages.cpp

527b07f

Co-authored-by: Taylor Foxhall <[email protected]> Signed-off-by: Anton Pryakhin <[email protected]>

Address review pt2

eb72da3

Signed-off-by: Anton Pryakhin <[email protected]>

Merge branch 'main' into logdumper

93c314c

Signed-off-by: Anton Pryakhin <[email protected]>

waldgange requested a review from hallfox December 5, 2024 15:08

bmq-oss-ci bot reviewed Dec 5, 2024

View reviewed changes

hallfox approved these changes Dec 6, 2024

View reviewed changes

waldgange merged commit 90b398b into bloomberg:main Dec 6, 2024
35 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: Log dumper #445

Feat: Log dumper #445

waldgange commented Oct 7, 2024 •

edited

Loading

bmq-oss-ci bot left a comment

bmq-oss-ci bot left a comment

bmq-oss-ci bot left a comment

chrisbeard commented Oct 9, 2024

waldgange commented Oct 16, 2024

bmq-oss-ci bot left a comment

bmq-oss-ci bot left a comment

bmq-oss-ci bot left a comment

bmq-oss-ci bot left a comment

bmq-oss-ci bot left a comment

bmq-oss-ci bot left a comment

hallfox Nov 11, 2024

waldgange Nov 20, 2024

hallfox Nov 27, 2024

waldgange Dec 5, 2024

hallfox Nov 11, 2024

waldgange Nov 20, 2024

hallfox Nov 11, 2024

waldgange Nov 22, 2024

hallfox Nov 11, 2024

waldgange Nov 22, 2024

hallfox Nov 11, 2024

waldgange Nov 20, 2024

hallfox Nov 27, 2024

bmq-oss-ci bot left a comment

		// TODO: use obj.logDump() when the config is updated
		// .setRecordBufferSize(obj.logDump().recordBufferSize());

		// TODO: uncomment when setup of this parameter is uncommented
		// lc.bslsLogSeverityThreshold() = "error";

Feat: Log dumper #445

Feat: Log dumper #445

Conversation

waldgange commented Oct 7, 2024 • edited Loading

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

chrisbeard commented Oct 9, 2024

waldgange commented Oct 16, 2024

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

waldgange commented Oct 7, 2024 •

edited

Loading