udp transfer message lost on same host. message info: 50hz, size 133 bytes, platform: ubuntu 18.04

[gyyc233]

Is there an already existing issue for this?

• I have searched the existing issues

Expected behavior

• I defined the IdlResponse topic of including timestamp, index, thread_id and int_vec(push some info)
• On same host, used 5 process to publish IdlResponse with 50hz and used 1 process to subscribe this topic. At the same time recorded log of each publisher process and subscribe process. There are 5 log lifes of publisher and 1 log file of subscribe
• Compare the logs of subscriber and each publisher, check the message index.
• Check the latency and loss via log
  • latency = (timestamp of onTopicReceived) - (timestamp of publish)
  • loss: check the index of log files

Current behavior

• It takes more time for the subscriber to receive the first message

Subscriber matched: 1 topic: IdlResponse
=== IdlResponse === index: 0 , size: 133 bytes, thread_id: 547971010976 , cost(us): 12291
=== IdlResponse === index: 1 , size: 133 bytes, thread_id: 547971010976 , cost(us): 518
=== IdlResponse === index: 2 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1291
=== IdlResponse === index: 3 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1510
=== IdlResponse === index: 4 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1573
=== IdlResponse === index: 5 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1521

...

=== IdlResponse === index: 192 , size: 133 bytes, thread_id: 547971010976 , cost(us): 550
=== IdlResponse === index: 193 , size: 133 bytes, thread_id: 547971010976 , cost(us): 581
Subscriber matched: 2 topic: IdlResponse
=== IdlResponse === index: 194 , size: 133 bytes, thread_id: 547971010976 , cost(us): 730
=== IdlResponse === index: 195 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1072
=== IdlResponse === index: 196 , size: 133 bytes, thread_id: 547971010976 , cost(us): 2558
=== IdlResponse === index: 197 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1131
=== IdlResponse === index: 0 , size: 133 bytes, thread_id: 547652223392 , cost(us): 95492
=== IdlResponse === index: 1 , size: 133 bytes, thread_id: 547652223392 , cost(us): 75735
=== IdlResponse === index: 2 , size: 133 bytes, thread_id: 547652223392 , cost(us): 56703
=== IdlResponse === index: 3 , size: 133 bytes, thread_id: 547652223392 , cost(us): 36481

• After the subscriber matches the third publisher, some messages of this publisher are not printed on the subscriber
  • The error occurs, the first log of the third publisher is not index: 0 but index: 59

=== IdlResponse === index: 162 , size: 133 bytes, thread_id: 547652223392 , cost(us): 1461
=== IdlResponse === index: 355 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1445
=== IdlResponse === index: 163 , size: 133 bytes, thread_id: 547652223392 , cost(us): 2412
=== IdlResponse === index: 356 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1262
=== IdlResponse === index: 164 , size: 133 bytes, thread_id: 547652223392 , cost(us): 1942
=== IdlResponse === index: 357 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1152
Subscriber matched: 3 topic: IdlResponse
=== IdlResponse === index: 165 , size: 133 bytes, thread_id: 547652223392 , cost(us): 1287
=== IdlResponse === index: 358 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1372
=== IdlResponse === index: 166 , size: 133 bytes, thread_id: 547652223392 , cost(us): 1199
=== IdlResponse === index: 359 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1357
=== IdlResponse === index: 167 , size: 133 bytes, thread_id: 547652223392 , cost(us): 1452
=== IdlResponse === index: 360 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1243
=== IdlResponse === index: 168 , size: 133 bytes, thread_id: 547652223392 , cost(us): 1789
=== IdlResponse === index: 361 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1291
=== IdlResponse === index: 169 , size: 133 bytes, thread_id: 547652223392 , cost(us): 1449
=== IdlResponse === index: 59 , size: 133 bytes, thread_id: 548438143392 , cost(us): 188995
=== IdlResponse === index: 60 , size: 133 bytes, thread_id: 548438143392 , cost(us): 168658
=== IdlResponse === index: 61 , size: 133 bytes, thread_id: 548438143392 , cost(us): 148408
=== IdlResponse === index: 62 , size: 133 bytes, thread_id: 548438143392 , cost(us): 128079
=== IdlResponse === index: 63 , size: 133 bytes, thread_id: 548438143392 , cost(us): 111453
=== IdlResponse === index: 64 , size: 133 bytes, thread_id: 548438143392 , cost(us): 91199
=== IdlResponse === index: 65 , size: 133 bytes, thread_id: 548438143392 , cost(us): 70823
=== IdlResponse === index: 66 , size: 133 bytes, thread_id: 548438143392 , cost(us): 50525
=== IdlResponse === index: 67 , size: 133 bytes, thread_id: 548438143392 , cost(us): 29960
=== IdlResponse === index: 68 , size: 133 bytes, thread_id: 548438143392 , cost(us): 9643
=== IdlResponse === index: 362 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1298
=== IdlResponse === index: 170 , size: 133 bytes, thread_id: 547652223392 , cost(us): 2281
=== IdlResponse === index: 69 , size: 133 bytes, thread_id: 548438143392 , cost(us): 2201
=== IdlResponse === index: 363 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1224
=== IdlResponse === index: 171 , size: 133 bytes, thread_id: 547652223392 , cost(us): 2058
=== IdlResponse === index: 70 , size: 133 bytes, thread_id: 548438143392 , cost(us): 1732
=== IdlResponse === index: 364 , size: 133 bytes, thread_id: 547971010976 , cost(us): 1304

Steps to reproduce

• 5 publisher process and 1 subscriber

idl file

struct IdlResponse
{
    long long timestamp;
    sequence<double,5120000> int_vec;
    unsigned long index;
    string thread_id;
};

publisher code:

std::shared_ptr<IdlResponsePubSubType> type(new IdlResponsePubSubType);
std::string topic = "IdlResponse";

DDSPublisher pub(topic, type);
std::thread thread([&pub](){
    std::thread::id thread_id = std::this_thread::get_id();
    std::stringstream ss;
    ss << thread_id;
    std::string str_id = ss.str();
    uint32_t count = 0;
    std::vector<double> int_vec(10);
    IdlResponse ex;
    ex.int_vec(int_vec);
    ex.thread_id(str_id);

    while(1){
        ex.index(count);
        auto ms = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::system_clock::now().time_since_epoch());
        ex.timestamp(std::move(ms.count()));
        std::cout <<"pub IdlResponse"<<" , timestamp: "<<ms.count()<< ", thread id: " << ex.thread_id()<<", index: "<<ex.index()<<" , size: "<<ex.getCdrSerializedSize(ex)<< std::endl;

        pub.publish<IdlResponse>(ex);

        std::this_thread::sleep_for(std::chrono::milliseconds(20));
        count++;
    }
});
thread.join();

subscriber code:

void onTopicReceived(void* msg) override {
    IdlResponse* example = (IdlResponse*)msg;
    std::thread::id thread_id = std::this_thread::get_id();
    auto ms = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::system_clock::now().time_since_epoch());
    std::cout << "=== IdlResponse === index: " << example->index() << " , size: " << example->getCdrSerializedSize(*example) << " bytes, thread_id: " << example->thread_id() << " , cost(us): " << std::move(ms.count()) - example->timestamp() << std::endl;
};

• dds config

setting of fast dds

    // eprosima::fastrtps::ParticipantAttributes attr
    attr.rtps.builtin.discovery_config.discoveryProtocol = eprosima::fastrtps::rtps::DiscoveryProtocol_t::SIMPLE;
    attr.rtps.builtin.discovery_config.use_SIMPLE_EndpointDiscoveryProtocol = true;

    attr.rtps.builtin.discovery_config.m_simpleEDP.use_PublicationReaderANDSubscriptionWriter = true;
    attr.rtps.builtin.discovery_config.m_simpleEDP.use_PublicationWriterANDSubscriptionReader = true;

    attr.rtps.builtin.discovery_config.leaseDuration = 10;
    attr.rtps.builtin.discovery_config.leaseDuration_announcementperiod = eprosima::fastrtps::Duration_t(5, 0);
    attr.rtps.builtin.avoid_builtin_multicast = false;

udp setting

    // eprosima::fastrtps::ParticipantAttributes attr
    attr.rtps.useBuiltinTransports = false;

    auto udp_transport = std::make_shared<eprosima::fastrtps::rtps::UDPv4TransportDescriptor>();
    attr.rtps.userTransports.push_back(udp_transport);

qos

    // eprosima::fastrtps::PublisherAttributes attr;
    attr.topic.topicKind = eprosima::fastrtps::rtps::NO_KEY;
    attr.topic.historyQos.kind = eprosima::fastrtps::KEEP_LAST_HISTORY_QOS;
    attr.topic.historyQos.depth = 10;
    attr.topic.resourceLimitsQos.max_samples = 10;
    attr.topic.resourceLimitsQos.allocated_samples = 2;
    attr.times.heartbeatPeriod.seconds = 2;
    attr.times.heartbeatPeriod.nanosec = 200 * 1000 * 1000;
    attr.qos.m_reliability.kind = eprosima::fastrtps::RELIABLE_RELIABILITY_QOS;
    attr.qos.m_lifespan.duration = 60;

    // eprosima::fastrtps::SubscriberAttributes attr
    attr.topic.topicKind = eprosima::fastrtps::rtps::NO_KEY;
    attr.topic.historyQos.kind = eprosima::fastrtps::KEEP_LAST_HISTORY_QOS;
    attr.topic.historyQos.depth = 10;
    attr.topic.resourceLimitsQos.max_samples = 10;
    attr.topic.resourceLimitsQos.allocated_samples = 2;
    attr.qos.m_reliability.kind = eprosima::fastrtps::RELIABLE_RELIABILITY_QOS;
    attr.qos.m_durability.kind = eprosima::fastrtps::TRANSIENT_LOCAL_DURABILITY_QOS;
    attr.qos.m_lifespan.duration = 60;

Fast DDS version/commit

Fast-dds VERSION 2.5.0
Fast-cdr VERSION 1.0.23

Platform/Architecture

Other. Please specify in Additional context section.

Transport layer

UDPv4

Additional context

No response

XML configuration file

No response

Relevant log output

No response

Network traffic capture

No response

[JesusPoderoso]

Hi @gyyc233
Thank you for your feedback. Unfortunately, Fast DDS v2.5.0 is no longer supported.
Would you mind update to the last version 2.10.1?

Take into account that the discovery phase is asyncronous, regarding the increase of the amount of time for the subscriber to receive the first message issue. It usually happens that the subscriber has matched with the publisher when the publisher has not matched yet with the subscriber, or viceversa. In your reliable deployment, both cases can cause that initial message arrival delay.

About the message loss when subscriber matches with the third publisher, the proposed deployment results have a normal behaviour: the endpoints have been configured with history QoS KEEP LAST 10.
That explains both index missmatch and delay in the first messages received by the subscriber: when the publisher has already reached maximum samples (10), it starts to override the first one with the eleventh, the 2nd with the 12th... When they finally match, that message queue is beeing overrided constantly, so it takes some time until the queue gets empty (around your message 69, when yout timestamp returns to normal values).
If the desired behaviour is a full reliability with no message loss, then update the history QoS as KEEP ALL, or increase the history QoS depth and the resource limits QoS maximum samples to the desired amount.
Taking into account the 5 publisher - 1 subscriber deployment, it would be interesting the use of instances to include a logical separation and have, within the same Topic, several instances referring to the same data type.

Also, I am moving this ticket to the Support discussion forum as per Fast DDS Contributing guidelines. This is not a bug but a misconfiguration.

[gyyc233]

Hi @JesusPoderoso
Thanks for your reply, that's helpful.

According to the log, the publisher and subscriber match too long, causing the previous messages to be overwritten in the buffer.

By the way, I have another problem. The message info: 160057 bytes; frequency: 50 hz.

Set KEEP LAST 10, running 10 mins.

• From the publisher log shows that the message was published 27409 times, but the subscriber log show the messages was published 27396 times.
• The first message and last message was received, but lost some messages in between. 5 publisher - 1 subscriber deployment on same host.
• Then I increase the history QoS depth and the resource limits QoS maximum samples.(Change 10 to 30) This phenomenon disappears.

---

Search thread id to check the publish counts and subscribe counts

find the first and the last message

I would like to know details about message loss in this case. DDS version 2.5.0 Ubuntu 18.04