Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI only] Use RelWithDebInfo in Travis to get the callstacks #781

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

levb
Copy link
Collaborator

@levb levb commented Aug 6, 2024

Travis sanitize builds seem to still produce unique failures that are not appearing in GHA jobs. However, some were compiled with Release making call stack interpretation difficult (like, https://app.travis-ci.com/github/nats-io/nats.c/jobs/624866438). Use RelWithDebInfo to get the call stacks with symbols (hope it works in that environment, will see)

Copy link

codecov bot commented Aug 6, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.88%. Comparing base (1553d4a) to head (f7d6ea7).
Report is 9 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #781      +/-   ##
==========================================
+ Coverage   68.71%   68.88%   +0.17%     
==========================================
  Files          39       49      +10     
  Lines       15207    15219      +12     
  Branches     3143     3133      -10     
==========================================
+ Hits        10449    10484      +35     
+ Misses       1700     1687      -13     
+ Partials     3058     3048      -10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@levb
Copy link
Collaborator Author

levb commented Aug 8, 2024

As soon as I added RelWithDebInfo the failures seem to have disappeared, which shouldn't really happen as I understand RelWithDebInfo. I'll monitor this a little longer, and may add a matrix to see if the flags really matter for the failures.

@kozlovic
Copy link
Member

kozlovic commented Aug 8, 2024

@levb What I would do to test is modify a test to cause a data race or whatever the matrix is supposed to catch and see if the output is what you expect. Even a simple memory leak should produce an output, so I would take the first test to be executed and do a malloc without a free. You could do that in a separate commit that - once you verify that it works as expected - you would drop and "push force" without this commit. Makes sense?

@levb
Copy link
Collaborator Author

levb commented Aug 8, 2024

Right, I know that the call stack is displayed (on GHA, at least) - that's why I added these. But it appears that the races/whatever the warnings were just stopped happening with the symbol tables included. So I'd like to see if it really is the case. It is not a pressing issue, but I'd like to understand why is Travis still finding issues that GHA is not catching.

@kozlovic
Copy link
Member

kozlovic commented Aug 8, 2024

Right, I know that the call stack is displayed

Oh sorry, I thought you meant that you wanted to know if it would be displayed correctly :-)

but I'd like to understand why is Travis still finding issues that GHA is not catching.

My guess is timing. I believe that the Travis VMs (especially outside of the NATS server) are pretty basic/slow.

@kozlovic
Copy link
Member

kozlovic commented Aug 9, 2024

@levb I got one on GA, not sure if this is the one you were trying to catch?

024-08-09T16:27:11.4749004Z 215/302 Test #215: ReconnectFailsPendingRequest ............***Failed  Required regular expression not found. Regex=[ALL PASSED
2024-08-09T16:27:11.4751108Z ]  1.36 sec
2024-08-09T16:27:11.4752103Z Test running in VALGRIND mode.
2024-08-09T16:27:11.4822074Z Test server version: nats-server: v2.11.0-dev
2024-08-09T16:27:11.4823040Z �[0;34m
2024-08-09T16:27:11.4823548Z == ReconnectFailsPendingRequest ==
2024-08-09T16:27:11.4824790Z �[0;0m#01 Create options: 
2024-08-09T16:27:11.4825552Z �[0;32mPASSED�[0;0m
2024-08-09T16:27:11.4826039Z #02 Start server: 
2024-08-09T16:27:11.4826626Z �[0;32mPASSED�[0;0m
2024-08-09T16:27:11.4827240Z #03 Connect: 
2024-08-09T16:27:11.4827735Z �[0;32mPASSED�[0;0m
2024-08-09T16:27:11.4828663Z #04 Create service provider: 
2024-08-09T16:27:11.4829578Z �[0;32mPASSED�[0;0m
2024-08-09T16:27:11.4830160Z #05 Start thread that will stop server: 
2024-08-09T16:27:11.4830911Z �[0;32mPASSED�[0;0m
2024-08-09T16:27:11.4831550Z #06 Fails due to timeout: 
2024-08-09T16:27:11.4832161Z �[0;32mPASSED�[0;0m
2024-08-09T16:27:11.4832720Z #07 Create options: 
2024-08-09T16:27:11.4833499Z �[0;32mPASSED�[0;0m
2024-08-09T16:27:11.4834031Z #08 Start server: 
2024-08-09T16:27:11.4834602Z �[0;32mPASSED�[0;0m
2024-08-09T16:27:11.4835223Z #09 Connect: 
2024-08-09T16:27:11.4835743Z �[0;32mPASSED�[0;0m
2024-08-09T16:27:11.4836288Z #10 Create service provider: 
2024-08-09T16:27:11.4837110Z �[0;32mPASSED�[0;0m
2024-08-09T16:27:11.4837705Z #11 Start thread that will stop server: 
2024-08-09T16:27:11.4838810Z �[0;32mPASSED�[0;0m
2024-08-09T16:27:11.4839493Z #12 Fails due to disconnect: 
2024-08-09T16:27:11.4840107Z �[0;32mPASSED�[0;0m
2024-08-09T16:27:11.4840444Z 
2024-08-09T16:27:11.4840711Z =================================================================
2024-08-09T16:27:11.4841665Z ==8818==ERROR: LeakSanitizer: detected memory leaks
2024-08-09T16:27:11.4842229Z 
2024-08-09T16:27:11.4842595Z Direct leak of 106 byte(s) in 1 object(s) allocated from:
2024-08-09T16:27:11.4843881Z     #0 0x7f46512b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
2024-08-09T16:27:11.4845597Z     #1 0x55afc4c60c2d in natsMsg_createWithPadding /home/runner/work/nats.c/nats.c/src/msg.c:772
2024-08-09T16:27:11.4847002Z     #2 0x55afc4c612b1 in natsMsg_create /home/runner/work/nats.c/nats.c/src/msg.c:854
2024-08-09T16:27:11.4848396Z     #3 0x55afc4c79996 in _createControlMessage /home/runner/work/nats.c/nats.c/src/sub.c:66
2024-08-09T16:27:11.4850055Z     #4 0x55afc4c7b5aa in nats_createControlMessages /home/runner/work/nats.c/nats.c/src/sub.c:455
2024-08-09T16:27:11.4851580Z     #5 0x55afc4c7bc94 in natsSub_create /home/runner/work/nats.c/nats.c/src/sub.c:510
2024-08-09T16:27:11.4852970Z     #6 0x55afc4bfdf84 in natsConn_subscribeImpl /home/runner/work/nats.c/nats.c/src/conn.c:3044
2024-08-09T16:27:11.4854589Z     #7 0x55afc4bf5168 in natsConn_initResp /home/runner/work/nats.c/nats.c/src/conn.c:1470
2024-08-09T16:27:11.4856069Z     #8 0x55afc4c751f8 in natsConnection_RequestMsg /home/runner/work/nats.c/nats.c/src/pub.c:473
2024-08-09T16:27:11.4857610Z     #9 0x55afc4c75b20 in natsConnection_RequestString /home/runner/work/nats.c/nats.c/src/pub.c:557
2024-08-09T16:27:11.4859415Z     #10 0x55afc4adf854 in test_ReconnectFailsPendingRequest /home/runner/work/nats.c/nats.c/test/test.c:19954
2024-08-09T16:27:11.4860936Z     #11 0x55afc4be13f2 in main /home/runner/work/nats.c/nats.c/test/test.c:35871
2024-08-09T16:27:11.4862278Z     #12 0x7f4650429d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
2024-08-09T16:27:11.4862900Z 
2024-08-09T16:27:11.4863414Z Indirect leak of 106 byte(s) in 1 object(s) allocated from:
2024-08-09T16:27:11.4864711Z     #0 0x7f46512b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
2024-08-09T16:27:11.4866297Z     #1 0x55afc4c60c2d in natsMsg_createWithPadding /home/runner/work/nats.c/nats.c/src/msg.c:772
2024-08-09T16:27:11.4867899Z     #2 0x55afc4c612b1 in natsMsg_create /home/runner/work/nats.c/nats.c/src/msg.c:854
2024-08-09T16:27:11.4869288Z     #3 0x55afc4c79996 in _createControlMessage /home/runner/work/nats.c/nats.c/src/sub.c:66
2024-08-09T16:27:11.4870800Z     #4 0x55afc4c7b518 in nats_createControlMessages /home/runner/work/nats.c/nats.c/src/sub.c:453
2024-08-09T16:27:11.4872374Z     #5 0x55afc4c7bc94 in natsSub_create /home/runner/work/nats.c/nats.c/src/sub.c:510
2024-08-09T16:27:11.4873779Z     #6 0x55afc4bfdf84 in natsConn_subscribeImpl /home/runner/work/nats.c/nats.c/src/conn.c:3044
2024-08-09T16:27:11.4875517Z     #7 0x55afc4bf5168 in natsConn_initResp /home/runner/work/nats.c/nats.c/src/conn.c:1470
2024-08-09T16:27:11.4877106Z     #8 0x55afc4c751f8 in natsConnection_RequestMsg /home/runner/work/nats.c/nats.c/src/pub.c:473
2024-08-09T16:27:11.4879099Z     #9 0x55afc4c75b20 in natsConnection_RequestString /home/runner/work/nats.c/nats.c/src/pub.c:557
2024-08-09T16:27:11.4880784Z     #10 0x55afc4adf854 in test_ReconnectFailsPendingRequest /home/runner/work/nats.c/nats.c/test/test.c:19954
2024-08-09T16:27:11.4882447Z     #11 0x55afc4be13f2 in main /home/runner/work/nats.c/nats.c/test/test.c:35871
2024-08-09T16:27:11.4883690Z     #12 0x7f4650429d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
2024-08-09T16:27:11.4884353Z 
2024-08-09T16:27:11.4884738Z Indirect leak of 106 byte(s) in 1 object(s) allocated from:
2024-08-09T16:27:11.4886179Z     #0 0x7f46512b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
2024-08-09T16:27:11.4887731Z     #1 0x55afc4c60c2d in natsMsg_createWithPadding /home/runner/work/nats.c/nats.c/src/msg.c:772
2024-08-09T16:27:11.4889133Z     #2 0x55afc4c612b1 in natsMsg_create /home/runner/work/nats.c/nats.c/src/msg.c:854
2024-08-09T16:27:11.4890754Z     #3 0x55afc4c79996 in _createControlMessage /home/runner/work/nats.c/nats.c/src/sub.c:66
2024-08-09T16:27:11.4892453Z     #4 0x55afc4c7b561 in nats_createControlMessages /home/runner/work/nats.c/nats.c/src/sub.c:454
2024-08-09T16:27:11.4894218Z     #5 0x55afc4c7bc94 in natsSub_create /home/runner/work/nats.c/nats.c/src/sub.c:510
2024-08-09T16:27:11.4952946Z     #6 0x55afc4bfdf84 in natsConn_subscribeImpl /home/runner/work/nats.c/nats.c/src/conn.c:3044
2024-08-09T16:27:11.4954451Z     #7 0x55afc4bf5168 in natsConn_initResp /home/runner/work/nats.c/nats.c/src/conn.c:1470
2024-08-09T16:27:11.4956925Z     #8 0x55afc4c751f8 in natsConnection_RequestMsg /home/runner/work/nats.c/nats.c/src/pub.c:473
2024-08-09T16:27:11.4958593Z     #9 0x55afc4c75b20 in natsConnection_RequestString /home/runner/work/nats.c/nats.c/src/pub.c:557
2024-08-09T16:27:11.4960206Z     #10 0x55afc4adf854 in test_ReconnectFailsPendingRequest /home/runner/work/nats.c/nats.c/test/test.c:19954
2024-08-09T16:27:11.4961851Z     #11 0x55afc4be13f2 in main /home/runner/work/nats.c/nats.c/test/test.c:35871
2024-08-09T16:27:11.4964425Z     #12 0x7f4650429d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
2024-08-09T16:27:11.4965067Z 
2024-08-09T16:27:11.4966435Z Indirect leak of 106 byte(s) in 1 object(s) allocated from:
2024-08-09T16:27:11.4967835Z     #0 0x7f46512b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
2024-08-09T16:27:11.4969891Z     #1 0x55afc4c60c2d in natsMsg_createWithPadding /home/runner/work/nats.c/nats.c/src/msg.c:772
2024-08-09T16:27:11.4971262Z     #2 0x55afc4c612b1 in natsMsg_create /home/runner/work/nats.c/nats.c/src/msg.c:854
2024-08-09T16:27:11.4973199Z     #3 0x55afc4c79996 in _createControlMessage /home/runner/work/nats.c/nats.c/src/sub.c:66
2024-08-09T16:27:11.4974680Z     #4 0x55afc4c7b4cf in nats_createControlMessages /home/runner/work/nats.c/nats.c/src/sub.c:452
2024-08-09T16:27:11.4976066Z     #5 0x55afc4c7bc94 in natsSub_create /home/runner/work/nats.c/nats.c/src/sub.c:510
2024-08-09T16:27:11.4977667Z     #6 0x55afc4bfdf84 in natsConn_subscribeImpl /home/runner/work/nats.c/nats.c/src/conn.c:3044
2024-08-09T16:27:11.4979046Z     #7 0x55afc4bf5168 in natsConn_initResp /home/runner/work/nats.c/nats.c/src/conn.c:1470
2024-08-09T16:27:11.4980678Z     #8 0x55afc4c751f8 in natsConnection_RequestMsg /home/runner/work/nats.c/nats.c/src/pub.c:473
2024-08-09T16:27:11.4982591Z     #9 0x55afc4c75b20 in natsConnection_RequestString /home/runner/work/nats.c/nats.c/src/pub.c:557
2024-08-09T16:27:11.4984199Z     #10 0x55afc4adf854 in test_ReconnectFailsPendingRequest /home/runner/work/nats.c/nats.c/test/test.c:19954
2024-08-09T16:27:11.4985670Z     #11 0x55afc4be13f2 in main /home/runner/work/nats.c/nats.c/test/test.c:35871
2024-08-09T16:27:11.4987640Z     #12 0x7f4650429d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
2024-08-09T16:27:11.4988531Z 
2024-08-09T16:27:11.4991860Z Indirect leak of 106 byte(s) in 1 object(s) allocated from:
2024-08-09T16:27:11.5251630Z     #0 0x7f46512b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
2024-08-09T16:27:11.5253531Z     #1 0x55afc4c60c2d in natsMsg_createWithPadding /home/runner/work/nats.c/nats.c/src/msg.c:772
2024-08-09T16:27:11.5254722Z     #2 0x55afc4c612b1 in natsMsg_create /home/runner/work/nats.c/nats.c/src/msg.c:854
2024-08-09T16:27:11.5255897Z     #3 0x55afc4c79996 in _createControlMessage /home/runner/work/nats.c/nats.c/src/sub.c:66
2024-08-09T16:27:11.5257122Z     #4 0x55afc4c7b486 in nats_createControlMessages /home/runner/work/nats.c/nats.c/src/sub.c:451
2024-08-09T16:27:11.5258324Z     #5 0x55afc4c7bc94 in natsSub_create /home/runner/work/nats.c/nats.c/src/sub.c:510
2024-08-09T16:27:11.5259571Z     #6 0x55afc4bfdf84 in natsConn_subscribeImpl /home/runner/work/nats.c/nats.c/src/conn.c:3044
2024-08-09T16:27:11.5260866Z     #7 0x55afc4bf5168 in natsConn_initResp /home/runner/work/nats.c/nats.c/src/conn.c:1470
2024-08-09T16:27:11.5262112Z     #8 0x55afc4c751f8 in natsConnection_RequestMsg /home/runner/work/nats.c/nats.c/src/pub.c:473
2024-08-09T16:27:11.5263410Z     #9 0x55afc4c75b20 in natsConnection_RequestString /home/runner/work/nats.c/nats.c/src/pub.c:557
2024-08-09T16:27:11.5264909Z     #10 0x55afc4adf854 in test_ReconnectFailsPendingRequest /home/runner/work/nats.c/nats.c/test/test.c:19954
2024-08-09T16:27:11.5266306Z     #11 0x55afc4be13f2 in main /home/runner/work/nats.c/nats.c/test/test.c:35871
2024-08-09T16:27:11.5267652Z     #12 0x7f4650429d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
2024-08-09T16:27:11.5268222Z 
2024-08-09T16:27:11.5268694Z SUMMARY: AddressSanitizer: 530 byte(s) leaked in 5 allocation(s).
2024-08-09T16:27:11.5269359Z 

@levb
Copy link
Collaborator Author

levb commented Aug 9, 2024

@kozlovic thanks! This and one more that just happened point to natsSubscription_Destroy returning before the (own) thread terminates. I am looking into it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants