Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[eCAL 6] Linux - SHM Subscribers can appear connected but do not receive data #1914

Closed
DownerCase opened this issue Jan 17, 2025 · 6 comments

Comments

@DownerCase
Copy link
Contributor

DownerCase commented Jan 17, 2025

Status: Problem understood. When only a single subscriber for a topic is present, unsubscribing and resubscribing in less than the registration timeout period will result in the subscriber using a stale SHM file and be unable to receive new messages.


Extracted from #1880 (comment)

Problem Description

I found that with eCAL monitor applications when inspecting messages from topic A, switching to topic B, and then returning to topic A the application will no longer receive any messages, despite it indicating being connected.

How to reproduce

  • cmake --preset cli
  • cmake --build --preset cli
  • Run concurrently:
    • ./out/cli/build/bin/ecal_mon_tui
    • ./out/cli/build/bin/ecal_sample_person_snd
    • ./out/cli/build/bin/ecal_sample_minimal_snd
  • In the monitor:
    • Press enter to look at the messages from one topic - Works (messages visible after short delay)
    • Press down to switch to the other topic - Works (messages visible after short delay)
    • After less time than the registration timeout press up to go back to the original topic - Broken (messages never populate)
      • The monitor indicates 1 local connection for the topic but no data goes through

How did you get eCAL?

Built from source

Environment

  • eCAL: Master 5484c2b
  • OS: Linux x86_64 (Arch Linux)
  • gcc 14.2.1

eCAL System Information

Default config
@DownerCase
Copy link
Contributor Author

Ahaha, reproduced the problem in CI with a modified test case https://github.com/DownerCase/ecal/actions/runs/12828875683/job/35773823178#step:11:897 so not specific to my setup 😄

@DownerCase
Copy link
Contributor Author

Testing different config settings to narrow down the problem:

  • Force TCP publish and subscribe: ✔

  • Force UDP publish and subscribe: ✔

  • Force SHM publish and subscribe: ❌

  • SHM transport + shm registration: ❌

  • UDP transport + shm registration: ✔

  • SHM transport with:

    • zero copy mode: ❌
    • 1000 ack timeout: ❌
    • 2 memfile buffers: ❌

Problem appears to be with the SHM transport layer.

Log from good UDP transport
1737126168762 ms | dc-linux | SubscriberReconnection | 554513 | error | Could not load eCAL time sync module libecaltime-localtime.so
1737126168762 ms | dc-linux | SubscriberReconnection | 554513 | debug2 | foo::CPublisherImpl::Constructor
1737126168762 ms | dc-linux | SubscriberReconnection | 554513 | debug2 | bar::CPublisherImpl::Constructor
1737126168762 ms | dc-linux | SubscriberReconnection | 554513 | debug1 | foo::CSubscriberImpl::Constructor
1737126168762 ms | dc-linux | SubscriberReconnection | 554513 | debug2 | foo::CSubscriberImpl::SetReceiveCallback
1737126169762 ms | dc-linux | SubscriberReconnection | 554513 | debug2 | foo::CPublisherImpl::StartUdpLayer::ACTIVATED
1737126169762 ms | dc-linux | SubscriberReconnection | 554513 | debug4 | foo::CPublisherImpl::Register
1737126169762 ms | dc-linux | SubscriberReconnection | 554513 | debug2 | foo::CPublisherImpl::StartUdpLayer::WRITER_CREATED
1737126169762 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::ApplySubscriberRegistration
1737126169762 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CSubscriberImpl::ApplyPublisherRegistration
1737126169762 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::ApplySubscriberRegistration
1737126169762 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CSubscriberImpl::ApplyPublisherRegistration
1737126169762 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CSubscriberImpl::ApplyPublisherRegistration
1737126169763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp
1737126169763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp - SUCCESS
1737126169763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CSubscriberImpl::ApplySample
1737126169763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CSubscriberImpl::ApplySample::ReceiveCallback
1737126169763 ms | dc-linux | SubscriberReconnection | 554513 | debug1 | foo::CSubscriberImpl::Destructor
1737126169763 ms | dc-linux | SubscriberReconnection | 554513 | debug4 | foo::CSubscriberImpl::Unregister
1737126169763 ms | dc-linux | SubscriberReconnection | 554513 | debug1 | bar::CSubscriberImpl::Constructor
1737126169763 ms | dc-linux | SubscriberReconnection | 554513 | debug2 | bar::CSubscriberImpl::SetReceiveCallback
1737126169863 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp
1737126169863 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp - SUCCESS
1737126169963 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp
1737126169963 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp - SUCCESS
1737126170063 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp
1737126170063 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp - SUCCESS
1737126170163 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp
1737126170163 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp - SUCCESS
1737126170263 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp
1737126170264 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp - SUCCESS
1737126170364 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp
1737126170364 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp - SUCCESS
1737126170464 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp
1737126170464 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp - SUCCESS
1737126170564 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp
1737126170565 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp - SUCCESS
1737126170665 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp
1737126170665 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp - SUCCESS
1737126170763 ms | dc-linux | SubscriberReconnection | 554513 | debug2 | bar::CPublisherImpl::StartUdpLayer::ACTIVATED
1737126170763 ms | dc-linux | SubscriberReconnection | 554513 | debug4 | bar::CPublisherImpl::Register
1737126170763 ms | dc-linux | SubscriberReconnection | 554513 | debug2 | bar::CPublisherImpl::StartUdpLayer::WRITER_CREATED
1737126170763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::ApplySubscriberRegistration
1737126170763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CSubscriberImpl::ApplyPublisherRegistration
1737126170763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::ApplySubscriberUnregistration
1737126170763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::ApplySubscriberRegistration
1737126170763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CSubscriberImpl::ApplyPublisherRegistration
1737126170763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CSubscriberImpl::ApplyPublisherRegistration
1737126170764 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp
1737126170764 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp - SUCCESS
1737126170764 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CSubscriberImpl::ApplySample
1737126170764 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CSubscriberImpl::ApplySample::ReceiveCallback
1737126170764 ms | dc-linux | SubscriberReconnection | 554513 | debug1 | bar::CSubscriberImpl::Destructor
1737126170764 ms | dc-linux | SubscriberReconnection | 554513 | debug4 | bar::CSubscriberImpl::Unregister
1737126170764 ms | dc-linux | SubscriberReconnection | 554513 | debug1 | foo::CSubscriberImpl::Constructor
1737126170764 ms | dc-linux | SubscriberReconnection | 554513 | debug2 | foo::CSubscriberImpl::SetReceiveCallback
1737126170864 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp
1737126170864 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp - SUCCESS
1737126170964 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp
1737126170964 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp - SUCCESS
1737126171064 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp
1737126171064 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp - SUCCESS
1737126171165 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp
1737126171165 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp - SUCCESS
1737126171265 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp
1737126171265 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp - SUCCESS
1737126171365 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp
1737126171365 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp - SUCCESS
1737126171465 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp
1737126171465 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp - SUCCESS
1737126171566 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp
1737126171566 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp - SUCCESS
1737126171666 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp
1737126171666 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::Write::udp - SUCCESS
1737126171763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::ApplySubscriberRegistration
1737126171763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CSubscriberImpl::ApplyPublisherRegistration
1737126171763 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | bar::CPublisherImpl::ApplySubscriberUnregistration
1737126172764 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::ApplySubscriberRegistration
1737126172764 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CSubscriberImpl::ApplyPublisherRegistration
1737126172766 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp
1737126172766 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CPublisherImpl::Write::udp - SUCCESS
1737126172767 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CSubscriberImpl::ApplySample
1737126172767 ms | dc-linux | SubscriberReconnection | 554513 | debug3 | foo::CSubscriberImpl::ApplySample::ReceiveCallback
1737126172767 ms | dc-linux | SubscriberReconnection | 554513 | debug1 | foo::CSubscriberImpl::Destructor
1737126172767 ms | dc-linux | SubscriberReconnection | 554513 | debug4 | foo::CSubscriberImpl::Unregister
Log from bad SHM transport
1737126340388 ms | dc-linux | SubscriberReconnection | 564715 | error | Could not load eCAL time sync module libecaltime-localtime.so
1737126340388 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | foo::CPublisherImpl::Constructor
1737126340388 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | bar::CPublisherImpl::Constructor
1737126340389 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | foo::CSubscriberImpl::Constructor
1737126340389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | foo::CSubscriberImpl::SetReceiveCallback
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | foo::CPublisherImpl::StartShmLayer::ACTIVATED
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | CSyncMemoryFile::Create SUCCESS : ecal_e2c9c910
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | foo::CPublisherImpl::Register
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | foo::CPublisherImpl::StartShmLayer::WRITER_CREATED
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | CDataWriterSHM::ApplySubscription - Memory FileName: ecal_e2c9c910 to ProcessId 564715
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::ApplySubscriberRegistration
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CSubscriberImpl::ApplyPublisherRegistration
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | CDataWriterSHM::ApplySubscription - Memory FileName: ecal_e2c9c910 to ProcessId 564715
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::ApplySubscriberRegistration
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | CMemFileObserver ecal_e2c9c910 created
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | CMemFileObserver started.
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | CMemFileThreadPool::ObserveFile ecal_e2c9c910 added
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CSubscriberImpl::ApplyPublisherRegistration
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CSubscriberImpl::ApplyPublisherRegistration
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CSubscriberImpl::ApplySample
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CSubscriberImpl::ApplySample::ReceiveCallback
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | foo::CSubscriberImpl::Destructor
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | foo::CSubscriberImpl::Unregister
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | bar::CSubscriberImpl::Constructor
1737126341389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | bar::CSubscriberImpl::SetReceiveCallback
1737126341489 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126341489 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126341489 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126341489 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126341489 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126341589 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126341589 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126341590 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126341590 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126341590 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126341690 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126341690 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126341690 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126341690 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126341690 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126341790 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126341790 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126341790 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126341790 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126341790 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126341890 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126341890 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126341890 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126341890 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126341890 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126341990 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126341990 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126341990 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126341990 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126341990 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126342090 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126342090 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126342090 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126342090 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126342090 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126342190 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126342190 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126342190 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126342190 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126342190 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126342290 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126342290 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126342290 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126342290 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126342290 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | bar::CPublisherImpl::StartShmLayer::ACTIVATED
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | CSyncMemoryFile::Create SUCCESS : ecal_b65cf105
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | bar::CPublisherImpl::Register
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | bar::CPublisherImpl::StartShmLayer::WRITER_CREATED
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | CDataWriterSHM::ApplySubscription - Memory FileName: ecal_b65cf105 to ProcessId 564715
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::ApplySubscriberRegistration
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CSubscriberImpl::ApplyPublisherRegistration
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | CDataWriterSHM::RemoveSubscription - Memory FileName: ecal_e2c9c910 to ProcessId 564715
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::ApplySubscriberUnregistration
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | CDataWriterSHM::ApplySubscription - Memory FileName: ecal_b65cf105 to ProcessId 564715
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::ApplySubscriberRegistration
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | CMemFileObserver ecal_b65cf105 created
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | CMemFileObserver started.
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | CMemFileThreadPool::ObserveFile ecal_b65cf105 added
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CSubscriberImpl::ApplyPublisherRegistration
1737126342389 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CSubscriberImpl::ApplyPublisherRegistration
1737126342390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM
1737126342390 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126342390 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126342390 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126342390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM - SUCCESS
1737126342390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CSubscriberImpl::ApplySample
1737126342390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CSubscriberImpl::ApplySample::ReceiveCallback
1737126342390 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | bar::CSubscriberImpl::Destructor
1737126342390 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | bar::CSubscriberImpl::Unregister
1737126342390 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | foo::CSubscriberImpl::Constructor
1737126342390 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | foo::CSubscriberImpl::SetReceiveCallback
1737126342490 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM
1737126342490 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126342490 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126342490 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126342490 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM - SUCCESS
1737126342590 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM
1737126342590 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126342590 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126342590 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126342590 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM - SUCCESS
1737126342690 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM
1737126342690 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126342690 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126342690 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126342690 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM - SUCCESS
1737126342790 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM
1737126342790 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126342790 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126342790 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126342790 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM - SUCCESS
1737126342891 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM
1737126342891 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126342891 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126342891 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126342891 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM - SUCCESS
1737126342991 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM
1737126342991 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126342991 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126342991 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126342991 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM - SUCCESS
1737126343091 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM
1737126343091 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126343091 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126343091 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126343091 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM - SUCCESS
1737126343191 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM
1737126343191 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126343191 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126343191 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126343191 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM - SUCCESS
1737126343291 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM
1737126343291 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126343291 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126343291 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126343291 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::Write::SHM - SUCCESS
1737126343390 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | CDataWriterSHM::ApplySubscription - Memory FileName: ecal_e2c9c910 to ProcessId 564715
1737126343390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::ApplySubscriberRegistration
1737126343390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CSubscriberImpl::ApplyPublisherRegistration
1737126343390 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | CDataWriterSHM::RemoveSubscription - Memory FileName: ecal_b65cf105 to ProcessId 564715
1737126343390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | bar::CPublisherImpl::ApplySubscriberUnregistration
1737126344390 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | CDataWriterSHM::ApplySubscription - Memory FileName: ecal_e2c9c910 to ProcessId 564715
1737126344390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::ApplySubscriberRegistration
1737126344390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CSubscriberImpl::ApplyPublisherRegistration
1737126344392 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126344392 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126344392 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126344392 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126344392 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126344492 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126344492 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126344492 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126344492 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126344492 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126344592 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126344592 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126344592 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126344592 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126344592 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126344692 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126344692 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126344692 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126344692 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126344692 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126344792 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126344792 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126344792 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126344793 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126344793 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126344893 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126344893 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126344893 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126344893 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126344893 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126344993 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126344993 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126344993 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126344993 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126344993 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126345093 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126345093 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126345093 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126345093 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126345093 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126345193 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126345193 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126345193 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126345193 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126345193 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126345293 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126345293 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126345293 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126345293 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126345293 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126345390 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | CDataWriterSHM::ApplySubscription - Memory FileName: ecal_e2c9c910 to ProcessId 564715
1737126345390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::ApplySubscriberRegistration
1737126345390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CSubscriberImpl::ApplyPublisherRegistration
1737126345393 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126345393 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126345393 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126345393 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126345393 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126345493 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126345493 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126345493 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126345493 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126345493 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126345593 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126345593 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126345593 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126345593 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126345593 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126345693 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126345693 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126345693 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126345693 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126345693 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126345793 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126345793 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126345793 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126345793 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126345793 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126345893 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126345893 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126345893 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126345893 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126345893 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126345993 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126345993 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126345993 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126345993 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126345993 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126346094 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126346094 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126346094 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126346094 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126346094 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126346194 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126346194 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126346194 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126346194 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126346194 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126346294 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126346294 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126346294 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126346294 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126346294 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126346390 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | CDataWriterSHM::ApplySubscription - Memory FileName: ecal_e2c9c910 to ProcessId 564715
1737126346390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::ApplySubscriberRegistration
1737126346390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CSubscriberImpl::ApplyPublisherRegistration
1737126346394 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126346394 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126346394 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126346394 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126346394 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126346494 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126346494 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126346494 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126346494 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126346494 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126346594 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126346594 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126346594 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126346594 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126346594 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126346694 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126346694 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126346694 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126346694 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126346694 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126346794 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126346794 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126346794 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126346794 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126346794 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126346894 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126346894 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126346894 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126346895 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126346895 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126346995 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126346995 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126346995 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126346995 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126346995 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126347095 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126347095 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126347095 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126347095 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126347095 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126347195 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126347195 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126347195 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126347195 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126347195 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126347295 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM
1737126347295 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write
1737126347295 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::SignalWritten
1737126347295 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | ecal_::CSyncMemoryFile::Write - SUCCESS : 11 Bytes written
1737126347295 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::Write::SHM - SUCCESS
1737126347390 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | CDataWriterSHM::ApplySubscription - Memory FileName: ecal_e2c9c910 to ProcessId 564715
1737126347390 ms | dc-linux | SubscriberReconnection | 564715 | debug3 | foo::CPublisherImpl::ApplySubscriberRegistration
1737126347390 ms | dc-linux | SubscriberReconnection | 564715 | debug1 | foo::CSubscriberImpl::Destructor
1737126347390 ms | dc-linux | SubscriberReconnection | 564715 | debug4 | foo::CSubscriberImpl::Unregister
1737126347395 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | CMemFileObserver ecal_b65cf105 stopped
1737126347395 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | CMemFileObserver ecal_e2c9c910 stopped
1737126347395 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | CMemFileObserver  destroyed
1737126347395 ms | dc-linux | SubscriberReconnection | 564715 | debug2 | CMemFileObserver  destroyed

Nothing leaps out when diffing the two, the structure of the logs appears functionally identical.

@DownerCase DownerCase changed the title [eCAL 6] Linux - Subscribers can become connected but do not receive data [eCAL 6] Linux - SHM Subscribers can appear connected but do not receive data Jan 17, 2025
@DownerCase
Copy link
Contributor Author

Didn't get too close with this. I could see that in the zombie case sync events were being sent but somehow not received on the other end...

Address and undefined behaviour sanitizers didn't report anything but thread sanitizer was extraordinarily upset. I'll work through the various synchronization errors in the vain hope one of them will help...

@DownerCase
Copy link
Contributor Author

Ok, I've found that disabling the call to shm_unlink lets the test pass.

return(::shm_unlink(event_name_));

What is happening is that:

  1. New SHM file is created by publisher with ownership
  2. CMemFileObserver opens the newly created event
  3. Subscriber is destroyed, DataWriterSHM::RemoveSubscription is fired
  4. The data writer has no subscribers so it unlinks the SHM memory file
  5. The CMemFileObserver doesn't timeout and close the previously opened event
  6. A new subscriber comes along and wants to resubscribe to the original topic
  7. The original CMemFileObserver (which points to the old SHM file) is reused
  8. The publisher and subscriber are using different synchronization objects.
  9. DEADLOCK: The publisher will keep triggering ApplyPublisherRegistration which resets the memfile observer timeout...

@KerstinKeller
Copy link
Contributor

Thanks for debugging into this issue so much.
Without doubt, we need to have another round of cleanup in the SHM communication.

Maybe we should take a step back, and see what kind of behavior we we would like to see.
So in general, I see two solutions to the problem:

  1. SHM files for a publisher stay open / linked until the process is closed, e.g. unlinking will be triggered when destroying the publisher. This way the publisher will just pause writing to a given memory file, when there is no longer a subscriber for it. Memory file / synchronization events stay valid when a new subscriber reconnects.

  2. Whenever there are no more subscribers, the connection is resetted. When there is again a new subscriber interested in a given topic, a new communication will be established, meaning creating a new filename for memory file and events and communicating that to the subscribers.

I guess there are pros / cons to both approaches.

@KerstinKeller
Copy link
Contributor

Preliminary fixed by #1943.
Adress again when separating SHM transport layer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants