Try to bring back symbols on backtraces on linux

[kiburtse]

What, How & Why?

☑️ ToDos

• 📝 Changelog update
• 🚦 Tests (or not relevant)
• C-API, if public C++ API changed.

[kiburtse]

Ok.. On macos everything is printed fine with manually triggered assertion in sync tests, but on linux now some backtraces are pretty complete:

[2023/11/02 17:51:13.009] 2: ../src/realm/sync/noinst/client_history_impl.cpp:1196: [realm-core-13.23.2] Assertion failed: false with (ref, version) =  [3120, 5]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(+0x1260748) [0xaaaab9666748]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(+0x1260950) [0xaaaab9666950]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(_ZNSt19_Optional_base_implIN5realm4sync13ClientHistory6ArraysESt14_Optional_baseIS3_Lb0ELb0EEE6_M_getEv+0) [0xaaaab91b187c]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(_ZN5realm4sync13ClientHistory27update_from_ref_and_versionEmm+0x154) [0xaaaab91aed68]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(_ZN5realm4sync13ClientHistory18update_from_parentEm+0x30) [0xaaaab91aefc4]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(_ZNK5realm5_impl7History14ensure_updatedEm+0x44) [0xaaaab9197a44]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(_ZN5realm4sync17ClientReplication17prepare_changesetEPKcmm+0x40) [0xaaaab91aa38c]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(_ZN5realm11Replication14prepare_commitEm+0x60) [0xaaaab95864a8]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(_ZN5realm2DB9do_commitERNS_11TransactionEb+0x104) [0xaaaab94a4a04]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(_ZN5realm11Transaction6commitEv+0xe0) [0xaaaab95e4f78]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(_ZN5realm4sync50Realm_UnitTest__Sync_PendingBootstrapStoreBatching8test_runEv+0x940) [0xaaaab8fd1230]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(_ZN5realm9test_util9unit_test12RegisterTestINS_4sync50Realm_UnitTest__Sync_PendingBootstrapStoreBatchingEE8run_testERNS1_11TestContextE+0x34) [0xaaaab8fd2f94]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(_ZN5realm9test_util9unit_test8TestList17ThreadContextImpl3runENS2_17SharedContextImpl5EntryERNS_4util10UniqueLockE+0xc8) [0xaaaab90e3dcc]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(_ZN5realm9test_util9unit_test8TestList17ThreadContextImpl3runEv+0x108) [0xaaaab90e3ae0]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(+0xcdcb48) [0xaaaab90e2b48]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(+0xcdcbd8) [0xaaaab90e2bd8]
[2023/11/02 17:51:13.009] 2: /data/mci/93cf62926c706c0c1fb4c3d65799c144/realm-core/build/test/Debug/realm-sync-tests(+0xce191c) [0xaaaab90e791c]
[2023/11/02 17:51:13.009] 2: /lib/aarch64-linux-gnu/libpthread.so.0(+0x7624) [0xffffba8b5624]
[2023/11/02 17:51:13.009] 2: /lib/aarch64-linux-gnu/libc.so.6(+0xd149c) [0xffffba80c49c]

others are missing even the name of the test:

[2023/11/02 17:53:54.742] 2: ../src/realm/sync/noinst/client_history_impl.cpp:1196: [realm-core-13.23.2] Assertion failed: false with (ref, version) =  [792, 5]
[2023/11/02 17:53:54.742] 2: /data/mci/da5f7e218b033a47b6db34dffa4678a2/realm-core/build/test/RelWithDebInfo/realm-sync-tests() [0xbd0df7]
[2023/11/02 17:53:54.742] 2: /data/mci/da5f7e218b033a47b6db34dffa4678a2/realm-core/build/test/RelWithDebInfo/realm-sync-tests(_ZN5realm4util19terminate_with_infoEPKcS2_lS2_OSt16initializer_listINS0_9PrintableEE+0x141) [0xbd0d31]
[2023/11/02 17:53:54.742] 2: /data/mci/da5f7e218b033a47b6db34dffa4678a2/realm-core/build/test/RelWithDebInfo/realm-sync-tests() [0x8d7f9a]
[2023/11/02 17:53:54.742] 2: /data/mci/da5f7e218b033a47b6db34dffa4678a2/realm-core/build/test/RelWithDebInfo/realm-sync-tests(_ZN5realm4sync13ClientHistory27update_from_ref_and_versionEmm+0x122) [0x8d6d32]
[2023/11/02 17:53:54.742] 2: /data/mci/da5f7e218b033a47b6db34dffa4678a2/realm-core/build/test/RelWithDebInfo/realm-sync-tests(_ZN5realm4sync17ClientReplication17prepare_changesetEPKcmm+0x5b) [0x8d335b]
[2023/11/02 17:53:54.742] 2: /data/mci/da5f7e218b033a47b6db34dffa4678a2/realm-core/build/test/RelWithDebInfo/realm-sync-tests(_ZN5realm2DB9do_commitERNS_11TransactionEb+0xbe) [0xa9a2be]
[2023/11/02 17:53:54.742] 2: /data/mci/da5f7e218b033a47b6db34dffa4678a2/realm-core/build/test/RelWithDebInfo/realm-sync-tests(_ZN5realm11Transaction6commitEv+0x42) [0xb6bd82]
[2023/11/02 17:53:54.742] 2: /data/mci/da5f7e218b033a47b6db34dffa4678a2/realm-core/build/test/RelWithDebInfo/realm-sync-tests() [0x669d4a]
[2023/11/02 17:53:54.742] 2: /data/mci/da5f7e218b033a47b6db34dffa4678a2/realm-core/build/test/RelWithDebInfo/realm-sync-tests() [0x6685cd]
[2023/11/02 17:53:54.742] 2: /data/mci/da5f7e218b033a47b6db34dffa4678a2/realm-core/build/test/RelWithDebInfo/realm-sync-tests(_ZN5realm9test_util9unit_test8TestList17ThreadContextImpl3runENS2_17SharedContextImpl5EntryERNS_4util10UniqueLockE+0xf7) [0x887ce7]
[2023/11/02 17:53:54.742] 2: /data/mci/da5f7e218b033a47b6db34dffa4678a2/realm-core/build/test/RelWithDebInfo/realm-sync-tests(_ZN5realm9test_util9unit_test8TestList17ThreadContextImpl3runEv+0xb0) [0x887a20]
[2023/11/02 17:53:54.742] 2: /data/mci/da5f7e218b033a47b6db34dffa4678a2/realm-core/build/test/RelWithDebInfo/realm-sync-tests() [0x88b180]
[2023/11/02 17:53:54.742] 2: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f40707cf609]
[2023/11/02 17:53:54.742] 2: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f40706f4133]

Anyway, i think we should not define this flags at all, since not hiding symbols prevent compiler optimizations according to gcc docs, so every testing run will not actually test realm production code. @ironage what do you think we should do with this? Enable on a just a few builds specifically for testing?

EDIT: i should add still that even without symbols' names when on linux the build is RelWithDebInfo or just Debug addr2line -sfpC -e path/to/binary *some_address_from_stacktrace* resolves the source line just fine. So with an access to built artefacts we are fine without this fixes.

[coveralls-official]

Pull Request Test Coverage Report for Build kirill.burtsev_103

• 1 of 1 (100.0%) changed or added relevant line in 1 file are covered.
• 130 unchanged lines in 11 files lost coverage.
• Overall coverage decreased (-0.03%) to 91.667%

---

Files with Coverage Reduction	New Missed Lines	%
src/realm/parser/driver.cpp	2	85.17%
src/realm/table_view.cpp	2	94.18%
src/realm/util/serializer.cpp	2	90.03%
src/realm/sort_descriptor.cpp	3	93.7%
src/realm/sync/noinst/server/server.cpp	3	75.73%
src/realm/sync/instruction_applier.cpp	4	70.21%
src/realm/bplustree.cpp	7	75.72%
src/realm/sync/noinst/client_impl_base.cpp	7	85.31%
test/fuzz_group.cpp	13	54.4%
src/realm/util/file_mapper.cpp	35	69.01%

Totals
Change from base Build 1809:	-0.03%
Covered Lines:	230726
Relevant Lines:	251701

---

💛 - Coveralls

[ironage]

@kiburtse this is a great start and having some symbols is better than nothing. But I agree that there is some risk that we aren't actually testing production optimized code anymore. What do you think about running some sort of shell script on our linux evergreen jobs that greps the log file for stack traces and symbolocates them with addr2line ? Having it done on the job itself would make it so that we don't have to store the binaries and go do this ourselves if it happens.

[kiburtse]

@kiburtse this is a great start and having some symbols is better than nothing. But I agree that there is some risk that we aren't actually testing production optimized code anymore. What do you think about running some sort of shell script on our linux evergreen jobs that greps the log file for stack traces and symbolocates them with addr2line ? Having it done on the job itself would make it so that we don't have to store the binaries and go do this ourselves if it happens.

yeah.. should be separate script for ci builds. I'll try it in different pr. Seems like this one didn't trigger anything through multiple manual runs.