tbb的private_server与假阳性ThreadSanitizer数据竞赛

Question

我们经常收到虚假的ThreadSanitizer (tsan)数据竞争警告，但不一致。虽然众所周知，tsan可以发出虚假的积极警告，其中一些警告可能通过TSAN_OPTIONS环境变量加以抑制，但我们遇到的一类特殊警告似乎与英特尔的线程构建块(tbb)使用tbb::detail::r1::rml::private_server特别相关，如果我们能够更好地控制该private_server的停止，这类警告似乎是可以预防的。以下是在谷歌测试运行过程中遇到的这样一个假阳性tsan数据竞争警告：

WARNING: ThreadSanitizer: data race (pid=5244)
  Write of size 1 at 0x7ffda4d64fd8 by main thread:
    #0 std::shared_lock<std::shared_mutex>::shared_lock(std::shared_mutex&, std::defer_lock_t) /usr/local/foo-deps/20220316/include/c++/9.4.0/shared_mutex:639 (FooTest+0x68d162)
    #1 FooProxy::buildTranslationMapToOtherProxy(FooProxy*, std::vector<foo::StringOpInfo, std::allocator<foo::StringOpInfo> > const&) const /home/jenkins-slave/workspace/core-tsan-gcc/Foo/FooProxy.cpp:323 (FooTest+0x68d162)
    #2 FooProxy_BuildTranslationMapToPartialOverlapProxy_Test::TestBody() /home/jenkins-slave/workspace/core-tsan-gcc/Tests/FooTest.cpp:798 (FooTest+0x5c5284)
    #3 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:3968 (FooTest+0x62d798)
    #4 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:4004 (FooTest+0x62d798)
    #5 testing::Test::Run() /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:4043 (FooTest+0x618586)
    #6 testing::TestInfo::Run() /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:4219 (FooTest+0x6187d4)
    #7 testing::TestSuite::Run() /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:4351 (FooTest+0x618959)
    #8 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:6892 (FooTest+0x618e7e)
    #9 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:3968 (FooTest+0x62de38)
    #10 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:4004 (FooTest+0x62de38)
    #11 testing::UnitTest::Run() /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:6479 (FooTest+0x619440)
    #12 RUN_ALL_TESTS() /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gtest/gtest.h:11696 (FooTest+0x5b401a)
    #13 main /home/jenkins-slave/workspace/core-tsan-gcc/Tests/FooTest.cpp:974 (FooTest+0x5b401a)

  Previous read of size 8 at 0x7ffda4d64fd8 by thread T18:
    [failed to restore the stack]

  Location is stack of main thread.

  Location is global '<null>' at 0x000000000000 ([stack]+0x00000001efd8)

  Thread T18 (tid=5264, running) created by main thread at:
    #0 pthread_create ../../.././libsanitizer/tsan/tsan_interceptors.cc:964 (libtsan.so.0+0x2cd6b)
    #1 tbb::detail::r1::rml::private_server::wake_some(int) <null> (FooTest+0x8828ce)
    #2 tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter>(tbb::detail::d1::task*, tbb::detail::r1::external_waiter&) <null> (FooTest+0x88b1c2)
    #3 tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) <null> (FooTest+0x86e74c)
    #4 Foo::getStringViews() const /home/jenkins-slave/workspace/core-tsan-gcc/Foo/Foo.cpp:1869 (FooTest+0x63612c)
    #5 Foo_GetStringViews_Test::TestBody() /home/jenkins-slave/workspace/core-tsan-gcc/Tests/FooTest.cpp:141 (FooTest+0x5c625c)
    #6 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:3968 (FooTest+0x62d798)
    #7 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:4004 (FooTest+0x62d798)
    #8 testing::Test::Run() /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:4043 (FooTest+0x618586)
    #9 testing::TestInfo::Run() /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:4219 (FooTest+0x6187d4)
    #10 testing::TestSuite::Run() /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:4351 (FooTest+0x618959)
    #11 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:6892 (FooTest+0x618e7e)
    #12 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:3968 (FooTest+0x62de38)
    #13 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:4004 (FooTest+0x62de38)
    #14 testing::UnitTest::Run() /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gmock-gtest-all.cc:6479 (FooTest+0x619440)
    #15 RUN_ALL_TESTS() /home/jenkins-slave/workspace/core-tsan-gcc/ThirdParty/googletest/gtest/gtest.h:11696 (FooTest+0x5b401a)
    #16 main /home/jenkins-slave/workspace/core-tsan-gcc/Tests/FooTest.cpp:974 (FooTest+0x5b401a)

SUMMARY: ThreadSanitizer: data race /usr/local/foo-deps/20220316/include/c++/9.4.0/shared_mutex:639 in std::shared_lock<std::shared_mutex>::shared_lock(std::shared_mutex&, std::defer_lock_t)

(一些名字因匿名而被更改。)按时间顺序分列的事件摘要：

1. 运行谷歌测试Foo.GetStringViews (线程T18框架#5)
   • 在此测试期间，一个实例ta of tbb::task_arena调用ta.execute([&] { tbb::parallel_for(...); });。
   • 这似乎运行了tbb::detail::r1::rml::private_server::wake_some(int)，它产生了一个线程，在Google测试之间存活下来。

2. 运行Google测试FooProxy.BuildTranslationMapToPartialOverlapProxy (主线程框架#2)
   • 此测试写入到前面的测试读取的0x7ffda4d64fd8地址。

我们的TSAN_OPTIONS环境变量设置为

suppressions=/path/to/tsan.suppressions, history_size=7, second_deadlock_stack=1, halt_on_error=1

我们推测，错误的阳性数据竞赛警告是由以下三个主要因素造成的：

• 两个独立的测试一个接一个地同步运行，在这些测试中不可能进行数据竞争，但碰巧是从同一个内存地址读取/写或写/写到/写/写到相同的内存地址。
• 线程的堆栈之一超过最大history_size=7并报告[failed to restore the stack]。
• 第一个线程生成一个tbb::detail::r1::rml::private_server，该一直存活到第二个测试。

这是因为第一次测试的tbb::detail::r1::rml::private_server仍然与第二次测试同时进行，而第二次测试混淆了tsan，将其标记为数据竞赛。

问题(S)

如何在每个测试的开始或结束时杀死tbb::detail::r1::rml::private_server线程？

或者，如果这是不可能的，我们是否可以在我们的tsan.suppressions文件或TSAN_OPTIONS环境变量中添加一些东西来具体地抑制这个错误警告而不隐藏可能发生的真正的数据竞争？

Answer 1

为了在每次Google之后杀死tbb::detail::r1::rml::private_server，我们重写了测试夹具 TearDown()方法：

void TearDown() override {
  // Expected to kill tbb::detail::r1::rml::private_server after each test,
  // which can otherwise trigger false positive tsan data race warnings.
  auto handle = tbb::task_scheduler_handle::get();
  tbb::finalize(handle, std::nothrow_t{});
}

在我们的TBB版本中，我们还必须使用#define TBB_PREVIEW_WAITING_FOR_WORKERS和#include <tbb/global_control.h>。

帕维尔·昆巴瑟夫( Pavel Kumbrasev )对此表示赞赏。

Answer 2

您可以将Mach信号量替换为调度信号量，以抑制警告。

请参阅以下链接：信号量

您还可以创建一个抑制文件来指定抑制运行时标志https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions。

如果这有帮助，您可以在编译时应用这些设置：

-fsanitize=thread -fsanitize-blacklist=sanitizer-thread-suppressions.txt