Introducing performance_test benchmarks tool for eCAL. #660
Replies: 7 comments 17 replies
-
Thanks for sharing, looks like a cool project! At first I thought it would use rmw_ecal, but I checked the code and you are really doing raw-ecal there (which is good, as the performance may differ for obvious reasons). Do you have results of any kind that you can share? |
Beta Was this translation helpful? Give feedback.
-
test result by handos performance tunningcpu power-savingsudo nano /etc/default/grub
GRUB_CMDLINE_LINUX="nosmt processor.max_cstate=0 intel_idle.max_cstate=0"
sudo update-grub
sudo reboot -h now cpu performance-modesudo apt-get install cpufrequtils
# to get the highest freq
sudo cpufreq-set -g performance network-stack64MB sudo cpufreq-set -g performance
sudo sysctl -w net.ipv4.udp_mem="10240087380016777216"
sudo sysctl -w net.core.netdev_max_backlog="30000"
sudo sysctl -w net.core.rmem_max="67108864"
sudo sysctl -w net.core.wmem_max="67108864"
sudo sysctl -w net.core.rmem_default="67108864"
sudo sysctl -w net.core.wmem_default="67108864" swapsudo swapoff -a
sudo nano /etc/fstab
# comment the swap config resultthe unit is ms.
@FlorianReimold must be something wrong with my code, I will keep checking the ecal test program. test envDELL-3630 PC
|
Beta Was this translation helpful? Give feedback.
-
Hi @ZhenshengLee first of all thank you for all the effort - great ! I checked the eCAL binding of the performance test and there are at least two major issues from performance perspective. First issue is discussed previously, the message protocol needs some extra time to serialize and deserialize the content. But besides this there is another issue (and maybe we need to extend the eCAL API here). If you use the The only way to avoid copies on the receiver side for raw data is to use a callback. I adapted the eCAL performance interface to use a simple callback method and I bypassed Protobuf by using some ugly casting writing and reading the needed time stamp and id into a bytes char array. Can you please give it a try ? There is a lot to improve .. but just for getting an idea .. // Copyright 2022 zhenshenglee, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef COMMUNICATION_ABSTRACTIONS__ECAL_COMMUNICATOR_HPP_
#define COMMUNICATION_ABSTRACTIONS__ECAL_COMMUNICATOR_HPP_
#include <ecal/ecal.h>
#include <string>
#include <memory>
#include "communicator.hpp"
#include "resource_manager.hpp"
#include "../experiment_configuration/qos_abstraction.hpp"
namespace performance_test
{
/// Translates abstract QOS settings to specific QOS settings for eCAL.
class EcalQOSAdapter
{
public:
/**
* \brief Constructs the QOS adapter.
* \param qos The abstract QOS settings the adapter should use to derive the implementation specific QOS settings.
*/
explicit EcalQOSAdapter(const QOSAbstraction qos)
: m_qos(qos)
{}
/// Returns derived eCAL reliability setting from the stored abstract QOS setting.
inline eCAL::QOS::eQOSPolicy_Reliability reliability() const
{
if (m_qos.reliability == QOSAbstraction::Reliability::BEST_EFFORT) {
return eCAL::QOS::eQOSPolicy_Reliability::best_effort_reliability_qos;
} else if (m_qos.reliability == QOSAbstraction::Reliability::RELIABLE) {
return eCAL::QOS::eQOSPolicy_Reliability::reliable_reliability_qos;
} else {
throw std::runtime_error("Unsupported QOS!");
}
}
/// Returns derived eCAL history policy setting from the stored abstract QOS setting.
inline eCAL::QOS::eQOSPolicy_HistoryKind history_kind() const
{
if (m_qos.history_kind == QOSAbstraction::HistoryKind::KEEP_ALL) {
return eCAL::QOS::eQOSPolicy_HistoryKind::keep_all_history_qos;
} else if (m_qos.history_kind == QOSAbstraction::HistoryKind::KEEP_LAST) {
return eCAL::QOS::eQOSPolicy_HistoryKind::keep_last_history_qos;
} else {
throw std::runtime_error("Unsupported QOS!");
}
}
/// Returns derived eCAL history depth setting from the stored abstract QOS setting.
int32_t history_depth() const
{
if (m_qos.history_kind == QOSAbstraction::HistoryKind::KEEP_LAST) {
return static_cast<int32_t>(m_qos.history_depth);
} else if (m_qos.history_kind == QOSAbstraction::HistoryKind::KEEP_ALL) {
// Keep all, keeps all. No depth required, but setting to dummy value.
return 1;
} else {
throw std::runtime_error("Unsupported QOS!");
}
}
private:
const QOSAbstraction m_qos;
};
/**
* \brief The plugin for eCAL.
* \tparam Msg The msg type to use.
*/
template<class Msg>
class EcalCommunicator : public Communicator
{
public:
/// The data type to use.
using DataType = typename Msg::EcalType;
/// Constructor which takes a reference \param lock to the lock to use.
explicit EcalCommunicator(SpinLock & lock)
: Communicator(lock),
m_publisher(nullptr),
m_subscriber(nullptr)
{
auto hz = static_cast<double>(this->m_ec.rate());
auto period_ms = 1000.0 / hz;
this->m_timeout_ms = std::max(10 * period_ms, 100.0);
}
/**
* \brief Publishes the provided data.
*
* The first time this function is called it also creates the publisher.
* Further it updates all internal counters while running.
* \param data The data to publish.
* \param time The time to fill into the data field.
*/
void publish(std::int64_t time)
{
if (!m_publisher)
{
const EcalQOSAdapter qos(m_ec.qos());
ResourceManager::get().init_ecal_runtime();
eCAL::QOS::SWriterQOS writerQos;
writerQos.history_kind = qos.history_kind();
writerQos.reliability = qos.reliability();
writerQos.history_kind_depth = qos.history_depth();
m_publisher = std::unique_ptr<eCAL::CPublisher>(
new eCAL::CPublisher(m_ec.topic_name()));
// https://github.com/ZhenshengLee/performance_test/issues/1
m_publisher->SetQOS(writerQos);
// initialize message layout
init_msg();
}
// set number of publisher memory buffers to improve performance
// https://github.com/ZhenshengLee/performance_test/issues/1
m_publisher->ShmSetBufferCount(1);
if (m_ec.is_zero_copy_transfer())
{
// enable zero copy mode
m_publisher->ShmEnableZeroCopy(1);
}
else
{
// disable zero copy mode
m_publisher->ShmEnableZeroCopy(0);
}
lock();
write_msg(m_bytes.data(), time, next_sample_id());
increment_sent(); // We increment before publishing so we don't have to lock twice.
unlock();
m_publisher->Send(m_bytes.data(), m_bytes.size());
}
/**
* \brief Reads received data.
*
* The first time this function is called it also creates the subscriber.
* In detail this function:
* * Reads samples.
* * Verifies that the data arrived in the right order, chronologically and also
* consistent with the publishing order.
* * Counts received and lost samples.
* * Calculates the latency of the samples received and updates the statistics
accordingly.
*/
// subscriber callback function
void on_receive(const struct eCAL::SReceiveCallbackData* data_)
{
read_msg(static_cast<char*>(data_->buf));
gSetEvent(m_event);
}
void update_subscription()
{
if (!m_subscriber)
{
const EcalQOSAdapter qos(m_ec.qos());
ResourceManager::get().init_ecal_runtime();
eCAL::QOS::SReaderQOS ReaderQos;
ReaderQos.history_kind = qos.history_kind();
ReaderQos.reliability = qos.reliability();
ReaderQos.history_kind_depth = qos.history_depth();
m_subscriber = std::unique_ptr<eCAL::CSubscriber>(
new eCAL::CSubscriber(m_ec.topic_name()));
// https://github.com/ZhenshengLee/performance_test/issues/1
m_subscriber->SetQOS(ReaderQos);
m_subscriber->AddReceiveCallback(std::bind(&EcalCommunicator::on_receive, this, std::placeholders::_2));
eCAL::gOpenEvent(&m_event);
}
// did we get new receives ?
if(gWaitForEvent(m_event, m_timeout_ms))
{
lock();
if (m_prev_timestamp >= m_rec_time)
{
throw std::runtime_error(
"Data consistency violated. Received sample with not strictly "
"older timestamp. Time diff: " + std::to_string(
m_rec_time - m_prev_timestamp) + " Data Time: " +
std::to_string(m_rec_time)
);
}
if (m_ec.roundtrip_mode() == ExperimentConfiguration::RoundTripMode::RELAY)
{
unlock();
publish(m_rec_time);
lock();
}
else
{
m_prev_timestamp = m_rec_time;
update_lost_samples_counter(m_rec_id);
add_latency_to_statistics(m_rec_time);
increment_received();
}
unlock();
}
}
/// Returns the data received in bytes.
std::size_t data_received()
{
return num_received_samples() * sizeof(DataType);
}
private:
int m_timeout_ms = 0;
std::int64_t m_rec_time = 0;
std::uint64_t m_rec_id = 0;
std::vector<char> m_bytes;
std::unique_ptr<eCAL::CPublisher> m_publisher;
std::unique_ptr<eCAL::CSubscriber> m_subscriber;
eCAL::EventHandleT m_event;
void init_msg()
{
DataType msg;
auto num_elements = msg.msg_array_size();
m_bytes.resize(static_cast<size_t>(num_elements * 4));
}
void read_msg(char* data)
{
auto int64_ptr = reinterpret_cast<std::int64_t*>(data);
m_rec_time = *int64_ptr;
int64_ptr++;
auto uint64_ptr = reinterpret_cast<std::uint64_t*>(int64_ptr);
m_rec_id= *uint64_ptr;
}
void write_msg(char* data, std::int64_t time, std::uint64_t id)
{
auto int64_ptr = reinterpret_cast<std::int64_t*>(data);
*int64_ptr = time;
int64_ptr++;
auto uint64_ptr = reinterpret_cast<std::uint64_t*>(int64_ptr);
*uint64_ptr = id;
}
};
} // namespace performance_test
#endif // COMMUNICATION_ABSTRACTIONS__ECAL_COMMUNICATOR_HPP_ |
Beta Was this translation helpful? Give feedback.
-
I add some comments and made some cleanups- Please use that version for testing. On my machine I got an improvement of factor 20 for none zero copy and factor 35 for zero copy ! // Copyright 2022 zhenshenglee, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef COMMUNICATION_ABSTRACTIONS__ECAL_COMMUNICATOR_HPP_
#define COMMUNICATION_ABSTRACTIONS__ECAL_COMMUNICATOR_HPP_
#include <ecal/ecal.h>
#include <string>
#include <memory>
#include "communicator.hpp"
#include "resource_manager.hpp"
#include "../experiment_configuration/qos_abstraction.hpp"
namespace performance_test
{
/// Translates abstract QOS settings to specific QOS settings for eCAL.
class EcalQOSAdapter
{
public:
/**
* \brief Constructs the QOS adapter.
* \param qos The abstract QOS settings the adapter should use to derive the implementation specific QOS settings.
*/
explicit EcalQOSAdapter(const QOSAbstraction qos)
: m_qos(qos)
{}
/// Returns derived eCAL reliability setting from the stored abstract QOS setting.
inline eCAL::QOS::eQOSPolicy_Reliability reliability() const
{
if (m_qos.reliability == QOSAbstraction::Reliability::BEST_EFFORT) {
return eCAL::QOS::eQOSPolicy_Reliability::best_effort_reliability_qos;
} else if (m_qos.reliability == QOSAbstraction::Reliability::RELIABLE) {
return eCAL::QOS::eQOSPolicy_Reliability::reliable_reliability_qos;
} else {
throw std::runtime_error("Unsupported QOS!");
}
}
/// Returns derived eCAL history policy setting from the stored abstract QOS setting.
inline eCAL::QOS::eQOSPolicy_HistoryKind history_kind() const
{
if (m_qos.history_kind == QOSAbstraction::HistoryKind::KEEP_ALL) {
return eCAL::QOS::eQOSPolicy_HistoryKind::keep_all_history_qos;
} else if (m_qos.history_kind == QOSAbstraction::HistoryKind::KEEP_LAST) {
return eCAL::QOS::eQOSPolicy_HistoryKind::keep_last_history_qos;
} else {
throw std::runtime_error("Unsupported QOS!");
}
}
/// Returns derived eCAL history depth setting from the stored abstract QOS setting.
int32_t history_depth() const
{
if (m_qos.history_kind == QOSAbstraction::HistoryKind::KEEP_LAST) {
return static_cast<int32_t>(m_qos.history_depth);
} else if (m_qos.history_kind == QOSAbstraction::HistoryKind::KEEP_ALL) {
// Keep all, keeps all. No depth required, but setting to dummy value.
return 1;
} else {
throw std::runtime_error("Unsupported QOS!");
}
}
private:
const QOSAbstraction m_qos;
};
/**
* \brief The plugin for eCAL.
* \tparam Msg The msg type to use.
*/
template<class Msg>
class EcalCommunicator : public Communicator
{
public:
/// The data type to use.
using DataType = typename Msg::EcalType;
/// Constructor which takes a reference \param lock to the lock to use.
explicit EcalCommunicator(SpinLock & lock)
: Communicator(lock),
m_publisher(nullptr),
m_subscriber(nullptr)
{
auto hz = static_cast<double>(this->m_ec.rate());
auto period_ms = 1000.0 / hz;
this->m_timeout_ms = std::max(10 * period_ms, 100.0);
}
/**
* \brief Publishes the provided data.
*
* The first time this function is called it also creates the publisher.
* Further it updates all internal counters while running.
* \param data The data to publish.
* \param time The time to fill into the data field.
*/
void publish(std::int64_t time)
{
if (!m_publisher)
{
const EcalQOSAdapter qos(m_ec.qos());
ResourceManager::get().init_ecal_runtime();
eCAL::QOS::SWriterQOS writerQos;
writerQos.history_kind = qos.history_kind();
writerQos.reliability = qos.reliability();
writerQos.history_kind_depth = qos.history_depth();
// create publisher
m_publisher = std::unique_ptr<eCAL::CPublisher>(new eCAL::CPublisher(m_ec.topic_name()));
// set QOS
m_publisher->SetQOS(writerQos);
// set number of used publisher memory buffer
m_publisher->ShmSetBufferCount(1);
if (m_ec.is_zero_copy_transfer())
{
// enable zero copy mode
m_publisher->ShmEnableZeroCopy(1);
}
else
{
// disable zero copy mode
m_publisher->ShmEnableZeroCopy(0);
}
// initialize message layout
init_msg();
}
// write time and id into message
lock();
write_msg(m_bytes.data(), time, next_sample_id());
increment_sent(); // We increment before publishing so we don't have to lock twice.
unlock();
// publish message
m_publisher->Send(m_bytes.data(), m_bytes.size());
}
/**
* \brief Reads received data.
*
* The first time this function is called it also creates the subscriber.
* In detail this function:
* * Reads samples.
* * Verifies that the data arrived in the right order, chronologically and also
* consistent with the publishing order.
* * Counts received and lost samples.
* * Calculates the latency of the samples received and updates the statistics
accordingly.
*/
// subscriber callback function
void on_receive(const struct eCAL::SReceiveCallbackData* data_)
{
// read time and id out of the received buffer
read_msg(static_cast<char*>(data_->buf), m_rec_time, m_rec_id);
// signal update_subscription to process
gSetEvent(m_event);
}
void update_subscription()
{
if (!m_subscriber)
{
const EcalQOSAdapter qos(m_ec.qos());
ResourceManager::get().init_ecal_runtime();
eCAL::QOS::SReaderQOS ReaderQos;
ReaderQos.history_kind = qos.history_kind();
ReaderQos.reliability = qos.reliability();
ReaderQos.history_kind_depth = qos.history_depth();
// create subscriber
m_subscriber = std::unique_ptr<eCAL::CSubscriber>(new eCAL::CSubscriber(m_ec.topic_name()));
// set QOS
m_subscriber->SetQOS(ReaderQos);
// add receive callback
m_subscriber->AddReceiveCallback(std::bind(&EcalCommunicator::on_receive, this, std::placeholders::_2));
// create sync event
eCAL::gOpenEvent(&m_event);
}
// did we get new receives ?
if(gWaitForEvent(m_event, m_timeout_ms))
{
lock();
if (m_prev_timestamp >= m_rec_time)
{
throw std::runtime_error(
"Data consistency violated. Received sample with not strictly "
"older timestamp. Time diff: " + std::to_string(
m_rec_time - m_prev_timestamp) + " Data Time: " +
std::to_string(m_rec_time)
);
}
if (m_ec.roundtrip_mode() == ExperimentConfiguration::RoundTripMode::RELAY)
{
unlock();
publish(m_rec_time);
lock();
}
else
{
m_prev_timestamp = m_rec_time;
update_lost_samples_counter(m_rec_id);
add_latency_to_statistics(m_rec_time);
increment_received();
}
unlock();
}
}
/// Returns the data received in bytes.
std::size_t data_received()
{
return num_received_samples() * msg_size();
}
private:
int m_timeout_ms = 0;
std::int64_t m_rec_time = 0;
std::uint64_t m_rec_id = 0;
std::vector<char> m_bytes;
std::unique_ptr<eCAL::CPublisher> m_publisher;
std::unique_ptr<eCAL::CSubscriber> m_subscriber;
eCAL::EventHandleT m_event;
void init_msg()
{
m_bytes.resize(msg_size());
}
void read_msg(char* data, std::int64_t& time, std::uint64_t& id)
{
auto int64_ptr = reinterpret_cast<std::int64_t*>(data);
time = *int64_ptr;
int64_ptr++;
auto uint64_ptr = reinterpret_cast<std::uint64_t*>(int64_ptr);
id= *uint64_ptr;
}
void write_msg(char* data, std::int64_t time, std::uint64_t id)
{
auto int64_ptr = reinterpret_cast<std::int64_t*>(data);
*int64_ptr = time;
int64_ptr++;
auto uint64_ptr = reinterpret_cast<std::uint64_t*>(int64_ptr);
*uint64_ptr = id;
}
size_t msg_size()
{
DataType msg;
auto num_elements = msg.msg_array_size();
const auto single_element_size(4);
return(num_elements * single_element_size);
}
};
} // namespace performance_test
#endif // COMMUNICATION_ABSTRACTIONS__ECAL_COMMUNICATOR_HPP_ |
Beta Was this translation helpful? Give feedback.
-
I changed all proto2 definitions to proto3 ones now and used fixed types instead variant encoded ones. The ECAL_PROTO version is working with a callback logic now but stills makes a copy of the message in the callback (like the raw variant btw. too). This needs to adapted next. See this PR ZhenshengLee/performance_test#5. |
Beta Was this translation helpful? Give feedback.
-
Thinking it's time for supporting zero-copy for rclcpp in using rmw_ecal! Tracking this issue in https://github.com/continental/rmw_ecal/issues/45 |
Beta Was this translation helpful? Give feedback.
-
@ZhenshengLee I replaced the event logic with a conditional variable one. Now I hope the consistency issue is fixed. I could not reproduce it anymore. Please recheck. |
Beta Was this translation helpful? Give feedback.
-
Need a performance benchmarks tool?
Hi everyone, I just added ability of support of eCAL to a benchmarks tool for robotics middleware from apex.ai company, which is open source at
https://github.com/ZhenshengLee/performance_test
Ability with the performance_test
with this tool you can
use the tool like
eCAL support
-DPERFORMANCE_TEST_ECAL_ENABLED=ON
-c ECAL
--zero-copy
): yesps:
eCAL::protobuf::CPublisher
is used.usage
test result example
screenshot
limitations and issues
currently issues are found with eCAL at ZhenshengLee/performance_test#1 and ZhenshengLee/performance_test#2
related disscusions
#625
Beta Was this translation helpful? Give feedback.
All reactions