Skip to content

KnowledgePerformance

Jump to bottom Edit New page
James Edmondson edited this page Jul 9, 2018 · 24 revisions

Introduction

MADARA Knowledge Bases do not operate in isolation. Performance of distributed knowledge sharing between agents depends on operating system functions, compiler optimizations, and configuration of buffers and quality-of-service. In this wiki, we discuss some of the tools available in MADARA to gauge knowledge performance related to latency and throughput within a host (intrahost) and between hosts (interhost). To gain access to the tests mentioned in this wiki, you need to compile MADARA with the tests feature enabled with base_build.sh or the direct mwc.pl process.

Table of Contents

TLDR Summary

  • For intrahost performance, try to just use the same knowledge base between threads. The performance is orders of magnitude faster than using a network transport between processes on the operating system. This is true for all operating systems and architectures
  • For multi-process performance, the smaller the data packets, the more messages that can be transferred reliably between knowledge bases
  • Quality-of-service settings like TransportSettings::queue_length (the buffer size for the OS and transport layer to use) can be extremely important to performance. If possible, always try to use a queue_length that is big enough to hold at least 1s of max expected data throughput and possibly 5-10s if you want maximum throughput and reliability

Intrahost Performance

There are two major considerations for judging intrahost performance: 1) multi-threaded performance and 2) multi-processed performance. The former performance is mostly gated by time spent in OS critical sections but can also be affected by CPU load and memory latency. The latter is dictated and bottlenecked almost entirely by OS prioritization and handling of network protocols, file pipes, and sockets.

test_reasoning_throughput

The main test for multi-threaded performance can be found in $MADARA_ROOT/bin/test_reasoning_throughput. This test mainly tests function calls on the knowledge base and common data abstractions, such as Integer containers. Example final output for such a call is shown below.

Intrahost network_profiler

The $MADARA_ROOT/bin/network_profiler tool can be used for testing most supported knowledge base transports including UDP unicast, broadcast, multicast, ZeroMQ, and DDS. The tool comes with built-in help (--help or -h options) and can be run on inter-and intra-process communication between knowledge bases on multiple hosts.

To run network_profiler on the same host for intrahost tests, open two terminals and launch the tool in each window. At least one network_profiler should be id 0 (-i 0, which is the publisher and default id), and at least one network_profiler should be not zero (e.g., i 1, which is a subscriber). The publisher will publish data of a user-specified size and frequency (default is to publish as fast as possible). The subscriber will receive data and post latency and throughput information for the configured QoS. This tool is very valuable to understand performance.

Below are some example runs on an Ubuntu 16.04 Virtual Machine for intrahost testing.

Intrahost Multicast Performance Small

Publisher: $MADARA_ROOT/bin/network_profiler
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1

Receiving for 60 s on UDP Multicast transport
Test: SUCCESS
Settings:
  Transport type: UDP Multicast
  Data size: 128 B
  Test time: 60 s
Latency:
  Min: 6904 ns
  Avg: 1142090 ns
  Max: 31139447 ns
Throughput:
  Messages received: 1267771
  Message rate: 21129.5 packets/s
  Data received: 162274688 B
  Data rate: 2.70458e+06 B/s

Intrahost Multicast Performance Medium (64KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 64000
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1

Receiving for 60 s on UDP Multicast transport
Test: SUCCESS
Settings:
  Transport type: UDP Multicast
  Data size: 64000 B
  Test time: 60 s
Latency:
  Min: 68003 ns
  Avg: 1889606 ns
  Max: 7002278 ns
Throughput:
  Messages received: 29882
  Message rate: 498.033 packets/s
  Data received: 1912448000 B
  Data rate: 3.18741e+07 B/s

Intrahost Multicast Performance Large (500KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1

Receiving for 60 s on UDP Multicast transport
Test: SUCCESS
Settings:
  Transport type: UDP Multicast
  Data size: 500000 B
  Test time: 60 s
Latency:
  Min: 6348807 ns
  Avg: 16113732 ns
  Max: 20839996 ns
Throughput:
  Messages received: 3616
  Message rate: 60.2667 packets/s
  Data received: 1808000000 B
  Data rate: 3.01333e+07 B/s

Intrahost Multicast Performance Large (500KB) Deep (50MB buffer)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 -q 50000000
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000

Receiving for 60 s on UDP Multicast transport
Test: SUCCESS
Settings:
  Transport type: UDP Multicast
  Data size: 500000 B
  Test time: 60 s
Latency:
  Min: 4693015 ns
  Avg: 12484017 ns
  Max: 24725457 ns
Throughput:
  Messages received: 4662
  Message rate: 77.7 packets/s
  Data received: 2331000000 B
  Data rate: 3.885e+07 B/s

Intrahost Unicast Performance Small

Publisher: $MADARA_ROOT/bin/network_profiler -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -u 127.0.0.1:30001

Receiving for 60 s on UDP transport
Test: SUCCESS
Settings:
  Transport type: UDP
  Data size: 128 B
  Test time: 60 s
Latency:
  Min: 10708 ns
  Avg: 3291926 ns
  Max: 10518131 ns
Throughput:
  Messages received: 4691469
  Message rate: 78191.1 packets/s
  Data received: 600508032 B
  Data rate: 1.00085e+07 B/s

Intrahost Unicast Performance Medium (64KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 64000 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -u 127.0.0.1:30001

Receiving for 60 s on UDP transport
Test: SUCCESS
Settings:
  Transport type: UDP
  Data size: 64000 B
  Test time: 60 s
Latency:
  Min: 40491 ns
  Avg: 208246 ns
  Max: 10450103 ns
Throughput:
  Messages received: 1035884
  Message rate: 17264.7 packets/s
  Data received: 66296576000 B
  Data rate: 1.10494e+09 B/s

Intrahost Unicast Performance Large (500KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -u 127.0.0.1:30001

Receiving for 60 s on UDP transport
Test: SUCCESS
Settings:
  Transport type: UDP
  Data size: 500000 B
  Test time: 60 s
Latency:
  Min: 242553 ns
  Avg: 375276 ns
  Max: 5181377 ns
Throughput:
  Messages received: 205562
  Message rate: 3426.03 packets/s
  Data received: 102781000000 B
  Data rate: 1.71302e+09 B/s

Intrahost Unicast Performance Large (500KB) Deep (50MB buffer)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 -q 50000000 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 -u 127.0.0.1:30001

Receiving for 60 s on UDP transport
Test: SUCCESS
Settings:
  Transport type: UDP
  Data size: 500000 B
  Test time: 60 s
Latency:
  Min: 246819 ns
  Avg: 374523 ns
  Max: 5452111 ns
Throughput:
  Messages received: 208807
  Message rate: 3480.12 packets/s
  Data received: 104403500000 B
  Data rate: 1.74006e+09 B/s

Intrahost Summary

  • Both UDP unicast and multicast can be used for intraprocess communication, but unicast tends to be better latency and throughput due to the copy cost of multicast as implemented by the operating system
  • There is no real comparison between multi-threaded performance and networked multi-process performance. Use multi-threading with a single knowledge base wherever possible for maximum performance. This gets even more drastic in performance difference as you cross the UDP datagram boundary (64KB)

Interhost Performance

Interhost performance focuses on the capability of the operating system and network to handle knowledge sharing between knowledge bases on two or more hosts. Interhost performance is facilitated by knowledge transports such as UDP unicast, broadcast, multicast, DDS, and ZeroMQ.

Interhost network_profiler

The $MADARA_ROOT/bin/network_profiler tool can be used for testing most supported knowledge base transports including UDP unicast, broadcast, multicast, ZeroMQ, and DDS. The tool comes with built-in help (--help or -h options) and can be run on inter-and intra-process communication between knowledge bases on multiple hosts.

To run network_profiler on two hosts for intrahost tests, open one terminal on each host and launch the network_profiler tool in each terminal window. At least one network_profiler should be id 0 (-i 0, which is the publisher and default id), and at least one network_profiler should be not zero (e.g., i 1, which is a subscriber). The publisher will publish data of a user-specified size and frequency (default is to publish as fast as possible). The subscriber will receive data and post latency and throughput information for the configured QoS. This tool is very valuable to understand performance.

Below are some example runs on an Ubuntu 16.04 Virtual Machine for intrahost testing.

Add a custom footer
Add a custom sidebar
Clone this wiki locally