Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Threads may access destructed global DbgCtl objects on shutdown #11819

Open
JosiahWI opened this issue Oct 14, 2024 · 0 comments
Open

Threads may access destructed global DbgCtl objects on shutdown #11819

JosiahWI opened this issue Oct 14, 2024 · 0 comments
Labels
ASan Address Sanitizer TSan
Milestone

Comments

@JosiahWI
Copy link
Contributor

JosiahWI commented Oct 14, 2024

The issue was originally reported by TSan as a shutdown race. That report is easily reproducible, but I don't have a TSan build right now to do so. There were concerns it could be a false positive, but ASan happened to actually catch it in the wild on CI. This does not mean definitively that it isn't a false positive, but as TSan rarely has false positives (except in certain cases of which I believe this is not one) and both tools detected it, I think it's reasonable to presume this is a real race unless it can be proven otherwise. Here is the ASan report:

==6586==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000001178 at pc 0x0000004ab418 bp 0x7ff4f57e1cc0 sp 0x7ff4f57e1cb0
READ of size 1 at 0x604000001178 thread T1 ([ET_NET 0])
    #0 0x4ab417 in DbgCtl::on() const ../include/tsutil/DbgCtl.h:111
    #1 0x725570 in PollCont::do_poll(long) ../src/iocore/net/PollCont.cc:89
    #2 0x71d2bc in NetHandler::waitForActivity(long) ../src/iocore/net/NetHandler.cc:362
    #3 0xd4b8c2 in EThread::execute_regular() ../src/iocore/eventsystem/UnixEThread.cc:307
    #4 0xd4bddd in EThread::execute() ../src/iocore/eventsystem/UnixEThread.cc:358
    #5 0xd48a44 in spawn_thread_internal ../src/iocore/eventsystem/Thread.cc:75
    #6 0x7ff4f9db21c9 in start_thread (/lib64/libpthread.so.0+0x81c9)
    #7 0x7ff4f9a0d8d2 in __GI___clone (/lib64/libc.so.6+0x398d2)

This is likely very nontrivial to fix because of the difficulty of managing shutdown of global objects. Joining all threads before shutdown may be one possible fix.

@JosiahWI JosiahWI added ASan Address Sanitizer TSan labels Oct 14, 2024
@bryancall bryancall added this to the 10.1.0 milestone Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ASan Address Sanitizer TSan
Projects
None yet
Development

No branches or pull requests

2 participants