Feat[Storagetool]: List queue appid numrecords #534

bbpetukhov · 2024-11-29T17:54:57Z

NB: This PR depends on #531 and will be ready for review as soon as #531 is merged

Describe your changes
Added new functional to bmqstoragetool to list number of all types of records per queue.
Supported resolution of Queue Key and App Key.
Added extra cmd line arg --min-records-per-queue <threshold> to restrict displaying queues with low number of records (less than given threshold)

Testing performed

Manual testing with a few real journal file and corresponding CSL files.
with/without CSL file to make sure that Queue Keys resolution working.
Added a new Unit Test to check storage tool output

Additional context

Usage example:
./bmqstoragetool.tsk --summary --min-records-per-queue 1012012 --journal-file file.bmq_journal --csl-file file.bmq_csl

Sample Output (section related to this PR):

...
Number of records per Queue:
    Queue                 : 1234567890
    Total Records         : 333334
    Num Queue Op Records  : 1
    Num Message Records   : 111111
    Num Confirm Records   : 111111
    Num Delete Records    : 111111
    Queue                 : 1234567890A
    Total Records         : 1111481
    Num Queue Op Records  : 881
    Num Message Records   : 100
    Num Confirm Records   : 670
    Num Records Per App   : 1234567890B=80 1234567890C=90 1234567890D=100 1234567890E=100 1234567890F=100 12345678910=100 12345678911=100 
    Num Delete Records    : 100

Sample Output showcasing the resolution of Queue Key and App Key:

Run without CSL file:

Number of records per Queue:
    Queue Key             : E12EF21974
    Total Records         : 5
    Num Queue Op Records  : 1
    Num Message Records   : 1
    Num Confirm Records   : 2
    Num Records Per App   : 793E7A6FD3=1 D6B7D942A8=1
    Num Delete Records    : 1

Run with CSL file (i.e. --csl-file ./bmq_csl_20240930_090323_CF83E8E13A.bmq_csl):

Number of records per Queue:
    Queue Key             : E12EF21974
    Queue URI             : bmq://bmq.test.persistent.fanout/dima
    Total Records         : 5
    Num Queue Op Records  : 1
    Num Message Records   : 1
    Num Confirm Records   : 2
    Num Records Per App   : baz=1 foo=1
    Num Delete Records    : 1

Signed-off-by: Dmitrii Petukhov <[email protected]>

Signed-off-by: Aleksandr Ivanov <[email protected]>

Signed-off-by: Dmitrii Petukhov <[email protected]>

alexander-e1off · 2024-12-02T08:11:53Z

src/applications/bmqstoragetool/bmqstoragetool.m.cpp

+         "min number of records per queue for detailed info to be displayed",
+         balcl::TypeInfo(&arguments.d_minRecordsPerQueue),
+         balcl::OccurrenceInfo(
+             bsl::numeric_limits<bsls::Types::Int64>::max())},


Consider to use zero as default value. In most configs usually zero is used as default to turn off the feature.

I did try to use Uint64, but balcl::TypeInfo does not support it. So I had to switch to Int64

As we discussed in person I keep Int64 for cmd line argument and use Uint64 in Parameters class.

src/applications/bmqstoragetool/m_bmqstoragetool_journalfileprocessor.cpp

src/applications/bmqstoragetool/m_bmqstoragetool_parameters.h

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.h

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.cpp

Co-authored-by: Alexander Ivanov <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Signed-off-by: Dmitrii Petukhov <[email protected]>

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.h

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov · 2025-01-13T10:23:24Z

Merged with upstream main. Ready for review.

678098 · 2025-01-13T14:26:18Z

src/applications/bmqstoragetool/m_bmqstoragetool_journalfileprocessor.t.cpp

+    // Prepare expected output
+    bmqu::MemOutStream expectedStream(bmqtst::TestHelperUtil::allocator());
+    expectedStream << "5 message(s) found.\n";
+    bsl::vector<const char*> fields(bmqtst::TestHelperUtil::allocator());


Not needed to fix now, but I opened an issue for this thing
#564

alexander-e1off · 2025-01-13T15:56:46Z

src/applications/bmqstoragetool/m_bmqstoragetool_parameters.h

@@ -106,6 +106,9 @@ struct CommandLineArguments {
    bool d_confirmed;
    /// Show only messages, confirmed by some of the appId's
    bool d_partiallyConfirmed;
+    // Show only messages, confirmed by some of the appId's


Looks like comment duplication here and below. Consider to use /// as for other cases.

alexander-e1off · 2025-01-13T15:58:17Z

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.cpp

@@ -204,6 +204,22 @@ void outputFooter(bsl::ostream&                         ostream,
 // class SearchResult
 // ==================

+bool SearchResult::processQueueOpRecord(


SearchResult is an interface and has no implementation. Why do we need dummy implementation for these methods?

alexander-e1off · 2025-01-13T16:03:32Z

src/groups/bmq/bmqu/bmqu_operationchain.h

@@ -192,7 +192,7 @@

 #if BSLS_COMPILERFEATURES_SIMULATE_CPP11_FEATURES
 // Include version that can be compiled with C++03
-// Generated on Tue Oct 15 17:39:53 2024
+// Generated on Thu Jan  9 16:44:32 2025


It seems it is not needed to update these files.

bbpetukhov and others added 12 commits November 28, 2024 12:17

wip

4a3ab78

Signed-off-by: Dmitrii Petukhov <[email protected]>

Add support for Queue URI

dc981b7

Signed-off-by: Dmitrii Petukhov <[email protected]>

Support printer formatting

8111259

Signed-off-by: Dmitrii Petukhov <[email protected]>

Can calc records by type per queue

c9a7c23

Signed-off-by: Dmitrii Petukhov <[email protected]>

Calculate per records counts app key

4d091fe

Signed-off-by: Dmitrii Petukhov <[email protected]>

Add ""min records per queue" cmd line parameter

90ab10a

Signed-off-by: Dmitrii Petukhov <[email protected]>

Move printRecord methods to RecordPrinter namespace

fd373de

Signed-off-by: Aleksandr Ivanov <[email protected]>

Resolve AppKey to AppId

0757a2a

Signed-off-by: Dmitrii Petukhov <[email protected]>

Update README.md

5b99185

Signed-off-by: Dmitrii Petukhov <[email protected]>

Sort app ids ny num records in asc order

059aacd

Signed-off-by: Dmitrii Petukhov <[email protected]>

Add comments

621ca50

Signed-off-by: Dmitrii Petukhov <[email protected]>

Run clang-format

185194d

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov force-pushed the list-queue-appid-numrecords branch from 608ae0c to 185194d Compare November 29, 2024 18:16

alexander-e1off reviewed Dec 2, 2024

View reviewed changes

bbpetukhov and others added 6 commits December 2, 2024 12:42

Update src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.cpp

7f3fa10

Co-authored-by: Alexander Ivanov <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Update src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.cpp

5734e06

Co-authored-by: Alexander Ivanov <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Update src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.cpp

12dfb48

Co-authored-by: Alexander Ivanov <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Always print Queue Key in detailed queue information

55fdc36

Signed-off-by: Dmitrii Petukhov <[email protected]>

Refactor method to calc total records count

079d26c

Signed-off-by: Dmitrii Petukhov <[email protected]>

Prefer const_iterator

b0de035

Signed-off-by: Dmitrii Petukhov <[email protected]>

alexander-e1off reviewed Dec 2, 2024

View reviewed changes

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.h Outdated Show resolved Hide resolved

Make min-records-per-queue parameter unsigned

7295527

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov changed the title ~~List queue appid numrecords~~ Feat[Storagetool]: List queue appid numrecords Dec 2, 2024

Run clang-format

959f269

Signed-off-by: Dmitrii Petukhov <[email protected]>

pniedzielski self-assigned this Dec 2, 2024

Remove redundant Parameters default ctor

1ef7d9b

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov force-pushed the list-queue-appid-numrecords branch from 1d1fc97 to 1ef7d9b Compare December 2, 2024 16:50

Add unit test for queue detailed output

ff069a3

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov force-pushed the list-queue-appid-numrecords branch from 9d70563 to ff069a3 Compare December 3, 2024 13:26

bbpetukhov marked this pull request as ready for review December 3, 2024 17:46

bbpetukhov requested a review from a team as a code owner December 3, 2024 17:46

bbpetukhov added 2 commits December 5, 2024 14:16

Merge branch 'main' into list-queue-appid-numrecords

e565543

Signed-off-by: Dmitrii Petukhov <[email protected]>

Run clang-format

313e91e

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov force-pushed the list-queue-appid-numrecords branch from dd1a0a7 to 313e91e Compare December 5, 2024 13:05

bbpetukhov added 4 commits December 9, 2024 16:26

Merge branch 'main' into list-queue-appid-numrecords

15d034a

Fix unit tests

988cccf

Signed-off-by: Dmitrii Petukhov <[email protected]>

Merge branch 'main' into list-queue-appid-numrecords

5e9e56a

Signed-off-by: Dmitrii Petukhov <[email protected]>

Run clang-format

e1eb49a

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov force-pushed the list-queue-appid-numrecords branch from 74f5f11 to 988cccf Compare December 20, 2024 17:33

bbpetukhov added 2 commits January 9, 2025 16:25

Merge branch 'main' into list-queue-appid-numrecords

83463d6

Signed-off-by: Dmitrii Petukhov <[email protected]>

Resolve conflicts after merge

c3a2fc8

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov force-pushed the list-queue-appid-numrecords branch from 3633b88 to c3a2fc8 Compare January 9, 2025 15:10

bbpetukhov added 6 commits January 9, 2025 17:12

Run clang-format

bc28fd7

Signed-off-by: Dmitrii Petukhov <[email protected]>

Fix unit tests

8a085d4

Signed-off-by: Dmitrii Petukhov <[email protected]>

Merge branch 'main' into list-queue-appid-numrecords

9c04387

Signed-off-by: Dmitrii Petukhov <[email protected]>

Run clang format

4d1c251

Signed-off-by: Dmitrii Petukhov <[email protected]>

Merge branch 'main' into list-queue-appid-numrecords

28fc3f5

Refactor Parameters class - test creator helper function

e28074b

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov force-pushed the list-queue-appid-numrecords branch from 5b31e2b to e28074b Compare January 13, 2025 09:40

678098 reviewed Jan 13, 2025

View reviewed changes

alexander-e1off reviewed Jan 13, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat[Storagetool]: List queue appid numrecords #534

Feat[Storagetool]: List queue appid numrecords #534

bbpetukhov commented Nov 29, 2024 •

edited

Loading

alexander-e1off Dec 2, 2024

bbpetukhov Dec 2, 2024

bbpetukhov Dec 3, 2024

bbpetukhov commented Jan 13, 2025

678098 Jan 13, 2025

alexander-e1off Jan 13, 2025

alexander-e1off Jan 13, 2025

alexander-e1off Jan 13, 2025

Feat[Storagetool]: List queue appid numrecords #534

Are you sure you want to change the base?

Feat[Storagetool]: List queue appid numrecords #534

Conversation

bbpetukhov commented Nov 29, 2024 • edited Loading

alexander-e1off Dec 2, 2024

Choose a reason for hiding this comment

bbpetukhov Dec 2, 2024

Choose a reason for hiding this comment

bbpetukhov Dec 3, 2024

Choose a reason for hiding this comment

bbpetukhov commented Jan 13, 2025

678098 Jan 13, 2025

Choose a reason for hiding this comment

alexander-e1off Jan 13, 2025

Choose a reason for hiding this comment

alexander-e1off Jan 13, 2025

Choose a reason for hiding this comment

alexander-e1off Jan 13, 2025

Choose a reason for hiding this comment

bbpetukhov commented Nov 29, 2024 •

edited

Loading