-
Notifications
You must be signed in to change notification settings - Fork 19
KnowledgePerformance
MADARA Knowledge Bases do not operate in isolation. Performance of distributed knowledge sharing between agents depends on operating system functions, compiler optimizations, and configuration of buffers and quality-of-service. In this wiki, we discuss some of the tools available in MADARA to gauge knowledge performance related to latency and throughput within a host (intrahost) and between hosts (interhost). To gain access to the tests mentioned in this wiki, you need to compile MADARA with the tests feature enabled with base_build.sh or the direct mwc.pl process.
TOC
- For intrahost performance, try to just use the same knowledge base between threads. The performance is orders of magnitude faster than using a network transport between processes on the operating system. This is true for all operating systems and architectures
- For multi-process performance, the smaller the data packets, the more messages that can be transferred reliably between knowledge bases
- Quality-of-service settings like TransportSettings::queue_length (the buffer size for the OS and transport layer to use) can be extremely important to performance. If possible, always try to use a queue_length that is big enough to hold at least 1s of max expected data throughput and possibly 5-10s if you want maximum throughput and reliability
There are two major considerations for judging intrahost performance: 1) multi-threaded performance and 2) multi-processed performance. The former performance is mostly gated by time spent in OS critical sections but can also be affected by CPU load and memory latency. The latter is dictated and bottlenecked almost entirely by OS prioritization and handling of network protocols, file pipes, and sockets.
The main test for multi-threaded performance can be found in $MADARA_ROOT/bin/test_reasoning_throughput. This test mainly tests function calls on the knowledge base and common data abstractions, such as Integer containers. Example final output for such a call is shown below.
The $MADARA_ROOT/bin/network_profiler tool can be used for testing most supported knowledge base transports including UDP unicast, broadcast, multicast, ZeroMQ, and DDS. The tool comes with built-in help (--help or -h options) and can be run on inter-and intra-process communication between knowledge bases on multiple hosts.
To run network_profiler on the same host for intrahost tests, open two terminals and launch the tool in each window. At least one network_profiler should be id 0 (-i 0, which is the publisher and default id), and at least one network_profiler should be not zero (e.g., i 1, which is a subscriber). The publisher will publish data of a user-specified size and frequency (default is to publish as fast as possible). The subscriber will receive data and post latency and throughput information for the configured QoS. This tool is very valuable to understand performance.
Below are some example runs on an Ubuntu 16.04 Virtual Machine for intrahost testing.
Interhost performance focuses on the capability of the operating system and network to handle knowledge sharing between knowledge bases on two or more hosts. Interhost performance is facilitated by knowledge transports such as UDP unicast, broadcast, multicast, DDS, and ZeroMQ.
The $MADARA_ROOT/bin/network_profiler tool can be used for testing most supported knowledge base transports including UDP unicast, broadcast, multicast, ZeroMQ, and DDS. The tool comes with built-in help (--help or -h options) and can be run on inter-and intra-process communication between knowledge bases on multiple hosts.
To run network_profiler on two hosts for intrahost tests, open one terminal on each host and launch the network_profiler tool in each terminal window. At least one network_profiler should be id 0 (-i 0, which is the publisher and default id), and at least one network_profiler should be not zero (e.g., i 1, which is a subscriber). The publisher will publish data of a user-specified size and frequency (default is to publish as fast as possible). The subscriber will receive data and post latency and throughput information for the configured QoS. This tool is very valuable to understand performance.
Below are some example runs on an Ubuntu 16.04 Virtual Machine for intrahost testing.