Skip to content

KnowledgePerformance

Jump to bottom Edit New page
James Edmondson edited this page Oct 3, 2024 · 24 revisions

Architecture | Knowledge Base | Networking | Containers | RCW | Threads | Optimizations | KaRL | Encryption | Checkpointing | Knowledge Performance | Logging

Introduction

MADARA Knowledge Bases do not operate in isolation. Performance of distributed knowledge sharing between agents depends on operating system functions, compiler optimizations, and configuration of buffers and quality-of-service. In this wiki, we discuss some of the tools available in MADARA to gauge knowledge performance related to latency and throughput within a host (intrahost) and between hosts (interhost). To gain access to the tests mentioned in this wiki, you need to compile MADARA with the tests feature enabled with base_build.sh or the direct mwc.pl process.

Table of Contents

TLDR Summary

  • For intrahost performance, try to just use the same knowledge base between threads. The performance is orders of magnitude faster than using a network transport between processes on the operating system. This is true for all operating systems and architectures
  • For multi-process performance, the smaller the data packets, the more messages that can be transferred reliably between knowledge bases Data Size Message Performance Data
  • For multi-process performance, the fewer variables, the higher the throughput and lower the latency. Variables sent over the network have to be martialed/demartialed and also require string-based lookups in the KB on the read thread side Variable Scaling Message Performance Data
  • Quality-of-service settings like TransportSettings::queue_length (the buffer size for the OS and transport layer to use) can be extremely important to performance. If possible, always try to use a queue_length that is big enough to hold at least 1s of max expected data throughput and possibly 5-10s if you want maximum throughput and reliability

Spreadsheet with Charts

Intrahost Performance

There are two major considerations for judging intrahost performance: 1) multi-threaded performance and 2) multi-processed performance. The former performance is mostly gated by time spent in OS critical sections but can also be affected by CPU load and memory latency. The latter is dictated and bottlenecked almost entirely by OS prioritization and handling of network protocols, file pipes, and sockets.

test_reasoning_throughput

The main test for multi-threaded performance can be found in $MADARA_ROOT/bin/test_reasoning_throughput. This test mainly tests function calls on the knowledge base and common data abstractions, such as Integer containers. Example final output for such a call is shown below.

Command: $MADARA_ROOT/bin/test_reasoning_throughput

Average time taken per rule evaluation was:
=========================================================================
 KaRL: Simple Increments           		                   348 ns
 KaRL: Multiple Increments         		                    84 ns
 KaRL: Simple Ternary Increments   		                   381 ns
 KaRL: Multiple Ternary Increments 		                   110 ns
 KaRL: Compiled Simple Increments  		                   207 ns
 KaRL: Compiled Multiple Inc       		                    80 ns
 KaRL: Compiled Simple Tern Inc    		                   224 ns
 KaRL: Compiled Multiple Tern Inc  		                   103 ns
 KaRL: Compiled Single Assign      		                   195 ns
 KaRL: Compiled Multiple Assign    		                    81 ns
 KaRL: Extern Function Call        		                   158 ns
 KaRL: Compiled Extern Inc Func    		                   234 ns
 KaRL: Compiled Extern Multi Calls 		                   105 ns
 KaRL: Looped Simple Increments    		                   185 ns
 KaRL: Optimized Loop              		                     0 ns
 KaRL: Looped Simple Ternary Inc   		                   196 ns
 KaRL: Looped Multiple Ternary Inc 		                   197 ns
 KaRL: Get Variable Reference      		                    54 ns
 KaRL: Get Expanded Reference      		                   640 ns
 KaRL: Normal Set Operation        		                   200 ns
 KaRL: Variable Reference Set      		                   145 ns
 KaRL: Variables Inc Var Ref       		                   208 ns
 KaRL container: Assignment        		                    51 ns
 KaRL container: Increments        		                    62 ns
 KaRL staged container: Assignment 		                     0 ns
 KaRL staged container: Increments 		                     0 ns
 C++: Optimized Assignments        		                     1 ns
 C++: Optimized Increments         		                     0 ns
 C++: Optimized Ternary Increments 		                     0 ns
 C++: Virtual Increments           		                     2 ns
 C++: Volatile Assignments         		                     0 ns
 C++: Volatile Increments          		                     1 ns
 C++: Volatile Ternary Increments  		                     1 ns
 C++: STL Atomic Increments        		                     6 ns
 C++: STL Recursive Increments     		                    21 ns
 C++: STL Mutex Increments         		                    20 ns
=========================================================================


Hertz for each test with 100000 iterations * 10 tests was:
=========================================================================
 KaRL: Simple Increments           		                 2.87 mhz
 KaRL: Multiple Increments         		                11.82 mhz
 KaRL: Simple Ternary Increments   		                 2.62 mhz
 KaRL: Multiple Ternary Increments 		                 9.07 mhz
 KaRL: Compiled Simple Increments  		                 4.82 mhz
 KaRL: Compiled Multiple Inc       		                12.45 mhz
 KaRL: Compiled Simple Tern Inc    		                 4.45 mhz
 KaRL: Compiled Multiple Tern Inc  		                 9.65 mhz
 KaRL: Compiled Single Assign      		                 5.11 mhz
 KaRL: Compiled Multiple Assign    		                12.28 mhz
 KaRL: Extern Function Call        		                 6.29 mhz
 KaRL: Compiled Extern Inc Func    		                 4.27 mhz
 KaRL: Compiled Extern Multi Calls 		                 9.46 mhz
 KaRL: Looped Simple Increments    		                 5.38 mhz
 KaRL: Optimized Loop              		               109.77 ghz
 KaRL: Looped Simple Ternary Inc   		                 5.10 mhz
 KaRL: Looped Multiple Ternary Inc 		                 5.06 mhz
 KaRL: Get Variable Reference      		                18.43 mhz
 KaRL: Get Expanded Reference      		                 1.56 mhz
 KaRL: Normal Set Operation        		                 4.98 mhz
 KaRL: Variable Reference Set      		                 6.85 mhz
 KaRL: Variables Inc Var Ref       		                 4.80 mhz
 KaRL container: Assignment        		                19.28 mhz
 KaRL container: Increments        		                15.89 mhz
 KaRL staged container: Assignment 		                 2.77 ghz
 KaRL staged container: Increments 		                 2.85 ghz
 C++: Optimized Assignments        		               722.60 mhz
 C++: Optimized Increments         		                 2.89 ghz
 C++: Optimized Ternary Increments 		                 2.87 ghz
 C++: Virtual Increments           		               411.79 mhz
 C++: Volatile Assignments         		                 1.45 ghz
 C++: Volatile Increments          		               527.58 mhz
 C++: Volatile Ternary Increments  		               530.15 mhz
 C++: STL Atomic Increments        		               160.61 mhz
 C++: STL Recursive Increments     		                46.64 mhz
 C++: STL Mutex Increments         		                48.18 mhz
=========================================================================

Takeaway: Intrahost Multi-threading performance can be in the megahertz (1M+ operations per second), and can even be this high when accessing data with the shared_ptr system for large data structures. Multi-threading is the best possible way to hit throughput and latency needs in mission-critical systems.

Intrahost network_profiler

The $MADARA_ROOT/bin/network_profiler tool can be used for testing most supported knowledge base transports including UDP unicast, broadcast, multicast, ZeroMQ, and DDS. The tool comes with built-in help (--help or -h options) and can be run on inter-and intra-process communication between knowledge bases on multiple hosts.

To run network_profiler on the same host for intrahost tests, open two terminals and launch the tool in each window. At least one network_profiler should be id 0 (-i 0, which is the publisher and default id), and at least one network_profiler should be not zero (e.g., i 1, which is a subscriber). The publisher will publish data of a user-specified size and frequency (default is to publish as fast as possible). The subscriber will receive data and post latency and throughput information for the configured QoS. This tool is very valuable to understand performance.

Below are some example runs on an Ubuntu 16.04 dedicated host OS for intrahost testing. All of these tests are blasting tests with an uncontrolled loop that publishes as fast as possible, which tend to overwhelm OS mechanisms. We'll show some controlled publish tests with larger data sizes on another wiki.

Intrahost Multicast Performance Small

Publisher: $MADARA_ROOT/bin/network_profiler
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1

Receiving for 60 s on UDP Multicast transport
Test: SUCCESS
Settings:
  Transport type: UDP Multicast
  Buffer size: 1000000 B
  Hosts: 
    239.255.0.1:4150
  Data size: 128 B
  Test time: 60 s
Latency:
  Min: 5596 ns
  Avg: 11075 ns
  Max: 774445 ns
Throughput:
  Messages received: 575714
  Message rate: 9595.23 packets/s
  Data received: 73691392 B
  Data rate: 1.22819e+06 B/s

Intrahost Multicast Performance Medium (64KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 64000
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1

Receiving for 60 s on UDP Multicast transport
Test: SUCCESS
Settings:
  Transport type: UDP Multicast
  Buffer size: 1000000 B
  Hosts: 
    239.255.0.1:4150
  Data size: 64000 B
  Test time: 60 s
Latency:
  Min: 95092 ns
  Avg: 155792 ns
  Max: 577476 ns
Throughput:
  Messages received: 258
  Message rate: 4.3 packets/s
  Data received: 16512000 B
  Data rate: 275200 B/s

Intrahost Multicast Performance Large (500KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1

Receiving for 60 s on UDP Multicast transport
Subscriber received no data.
Test: FAIL.

Intrahost Multicast Performance Large (500KB) Deep (50MB buffer)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 -q 50000000
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000

Receiving for 60 s on UDP Multicast transport
Subscriber received no data.
Test: FAIL.

Note: Multicast is especially susceptible to failures in uncontrolled blasting tests. To get the most out of it, you really have to control your publication loop (i.e., you have to slow down the loop).

Intrahost Unicast Performance Small

Publisher: $MADARA_ROOT/bin/network_profiler -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -u 127.0.0.1:30001

Test: SUCCESS
Settings:
  Transport type: UDP
  Buffer size: 1000000 B
  Hosts: 
    127.0.0.1:30001
    127.0.0.1:30000
  Data size: 128 B
  Test time: 60 s
Latency:
  Min: 5381 ns
  Avg: 697858 ns
  Max: 3428485 ns
Throughput:
  Messages received: 9085170
  Message rate: 151420 packets/s
  Data received: 1162901760 B
  Data rate: 1.93817e+07 B/s

Intrahost Unicast Performance Medium (64KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 64000 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -u 127.0.0.1:30001

Test: SUCCESS
Settings:
  Transport type: UDP
  Buffer size: 1000000 B
  Hosts: 
    127.0.0.1:30001
    127.0.0.1:30000
  Data size: 64000 B
  Test time: 60 s
Latency:
  Min: 36468 ns
  Avg: 156824 ns
  Max: 1010158 ns
Throughput:
  Messages received: 1746246
  Message rate: 29104.1 packets/s
  Data received: 111759744000 B
  Data rate: 1.86266e+09 B/s

Intrahost Unicast Performance Large (500KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -u 127.0.0.1:30001

Receiving for 60 s on UDP transport
Test: SUCCESS
Settings:
  Transport type: UDP
  Buffer size: 1000000 B
  Hosts: 
    127.0.0.1:30001
    127.0.0.1:30000
  Data size: 500000 B
  Test time: 60 s
Latency:
  Min: 183187 ns
  Avg: 294327 ns
  Max: 820721 ns
Throughput:
  Messages received: 253591
  Message rate: 4226.52 packets/s
  Data received: 126795500000 B
  Data rate: 2.11326e+09 B/s

Intrahost Unicast Performance Large (500KB) Deep (50MB buffer)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 -q 50000000 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 -u 127.0.0.1:30001

Receiving for 60 s on UDP transport
Test: SUCCESS
Settings:
  Transport type: UDP
  Buffer size: 50000000 B
  Hosts: 
    127.0.0.1:30001
    127.0.0.1:30000
  Data size: 500000 B
  Test time: 60 s
Latency:
  Min: 183872 ns
  Avg: 298409 ns
  Max: 1111223 ns
Throughput:
  Messages received: 252466
  Message rate: 4207.77 packets/s
  Data received: 126233000000 B
  Data rate: 2.10388e+09 B/s

Intrahost ZeroMQ TCP Performance Small

Publisher: $MADARA_ROOT/bin/network_profiler --zmq tcp://127.0.0.1:30000 --zmq tcp://127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 --zmq tcp://127.0.0.1:30001 --zmq tcp://127.0.0.1:30000

Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 1000000 B
  Hosts: 
    tcp://127.0.0.1:30001
    tcp://127.0.0.1:30000
  Data size: 128 B
  Test time: 60 s
Latency:
  Min: 273699 ns
  Avg: 49443721 ns
  Max: 92948113 ns
Throughput:
  Messages received: 14806581
  Message rate: 246776 packets/s
  Data received: 1895242368 B
  Data rate: 3.15874e+07 B/s

Intrahost ZeroMQ TCP Performance Medium (64KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 64000 --zmq tcp://127.0.0.1:30000 --zmq tcp://127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 --zmq tcp://127.0.0.1:30001 --zmq tcp://127.0.0.1:30000

Receiving for 60 s on 0MQ transport
Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 1000000 B
  Hosts: 
    tcp://127.0.0.1:30001
    tcp://127.0.0.1:30000
  Data size: 64000 B
  Test time: 60 s
Latency:
  Min: 182407 ns
  Avg: 13598448 ns
  Max: 32058180 ns
Throughput:
  Messages received: 3028500
  Message rate: 50475 packets/s
  Data received: 193824000000 B
  Data rate: 3.2304e+09 B/s

Intrahost ZeroMQ TCP Performance Large (500KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 --zmq tcp://127.0.0.1:30000 --zmq tcp://127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 --zmq tcp://127.0.0.1:30001 --zmq tcp://127.0.0.1:30000

Receiving for 60 s on UDP transport
Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 1000000 B
  Hosts: 
    tcp://127.0.0.1:30001
    tcp://127.0.0.1:30000
  Data size: 500000 B
  Test time: 60 s
Latency:
  Min: 192157 ns
  Avg: 76076730 ns
  Max: 190742858 ns
Throughput:
  Messages received: 472732
  Message rate: 7878.87 packets/s
  Data received: 236366000000 B
  Data rate: 3.93943e+09 B/s

Intrahost ZeroMQ TCP Performance Large (500KB) Deep (50MB buffer)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 -q 50000000 --zmq tcp://127.0.0.1:30000 --zmq tcp://127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -q 50000000 -i 1 --zmq tcp://127.0.0.1:30001 --zmq tcp://127.0.0.1:30000

Receiving for 60 s on 0MQ transport
Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 50000000 B
  Hosts: 
    tcp://127.0.0.1:30001
    tcp://127.0.0.1:30000
  Data size: 500000 B
  Test time: 60 s
Latency:
  Min: 286858 ns
  Avg: 73344231 ns
  Max: 206109556 ns
Throughput:
  Messages received: 493678
  Message rate: 8227.97 packets/s
  Data received: 246839000000 B
  Data rate: 4.11398e+09 B/s

Intrahost ZeroMQ IPC Performance Small

Publisher: $MADARA_ROOT/bin/network_profiler --zmq ipc:///tmp/network_profiler_0_0 --zmq ipc:///tmp/network_profiler_0_1
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 --zmq ipc:///tmp/network_profiler_0_1 --zmq ipc:///tmp/network_profiler_0_0

Receiving for 60 s on 0MQ transport
Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 1000000 B
  Hosts: 
    ipc:///tmp/network_profiler_0_1
    ipc:///tmp/network_profiler_0_0
  Data size: 128 B
  Test time: 60 s
Latency:
  Min: 140928 ns
  Avg: 5542036 ns
  Max: 12139278 ns
Throughput:
  Messages received: 15918500
  Message rate: 265308 packets/s
  Data received: 2037568000 B
  Data rate: 3.39595e+07 B/s

Intrahost ZeroMQ IPC Performance Medium (64KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 64000 --zmq ipc:///tmp/network_profiler_1_0 --zmq ipc:///tmp/network_profiler_1_1
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 --zmq ipc:///tmp/network_profiler_1_1 --zmq ipc:///tmp/network_profiler_1_0

Receiving for 60 s on 0MQ transport
Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 1000000 B
  Hosts: 
    ipc:///tmp/network_profiler_4_1
    ipc:///tmp/network_profiler_4_0
  Data size: 64000 B
  Test time: 60 s
Latency:
  Min: 174203 ns
  Avg: 16401214 ns
  Max: 51432756 ns
Throughput:
  Messages received: 2648355
  Message rate: 44139.2 packets/s
  Data received: 169494720000 B
  Data rate: 2.82491e+09 B/s

Intrahost ZeroMQ IPC Performance Large (500KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 --zmq ipc:///tmp/network_profiler_2_0 --zmq ipc:///tmp/network_profiler_2_1
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 --zmq ipc:///tmp/network_profiler_2_1 --zmq ipc:///tmp/network_profiler_2_0

Receiving for 60 s on UDP transport
Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 1000000 B
  Hosts: 
    ipc:///tmp/network_profiler_5_1
    ipc:///tmp/network_profiler_5_0
  Data size: 500000 B
  Test time: 60 s
Latency:
  Min: 211384 ns
  Avg: 111801275 ns
  Max: 203103275 ns
Throughput:
  Messages received: 387729
  Message rate: 6462.15 packets/s
  Data received: 193864500000 B
  Data rate: 3.23108e+09 B/s

Intrahost ZeroMQ IPC Performance Large (500KB) Deep (50MB buffer)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 -q 50000000 --zmq ipc:///tmp/network_profiler_3_0 --zmq ipc:///tmp/network_profiler_3_1
Subscriber: $MADARA_ROOT/bin/network_profiler -q 50000000 -i 1 --zmq ipc:///tmp/network_profiler_3_1 --zmq ipc:///tmp/network_profiler_3_0

Receiving for 60 s on 0MQ transport
Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 50000000 B
  Hosts: 
    ipc:///tmp/network_profiler_6_1
    ipc:///tmp/network_profiler_6_0
  Data size: 500000 B
  Test time: 60 s
Latency:
  Min: 180383 ns
  Avg: 100392935 ns
  Max: 214295423 ns
Throughput:
  Messages received: 431500
  Message rate: 7191.67 packets/s
  Data received: 215750000000 B
  Data rate: 3.59583e+09 B/s

Intrahost Variable Scaling

The $MADARA_ROOT/bin/network_profiler tool can also be used to scale the number of variables to gauge the impact of scaling variables from 1 to N. This is important for helping developers determine the cost of martialing/demartialing and variable lookups in the readthreads of transports, which block the KB and hamper the efficiency of the transport in transferring large blocks of data.

Intrahost Multicast 500KB Deep 10 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 10
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000

Receiving for 60 s on UDP Multicast transport
Test: SUCCESS
Settings:
  Transport type: UDP Multicast
  Buffer size: 50000000 B
  Hosts: 
    239.255.0.1:4150
  Data size: 50000 B
  Test time: 60 s
  Num vars: 10
Latency:
  Min: 285442 ns
  Avg: 444743 ns
  Max: 1141665 ns
Throughput:
  Messages received: 70924
  Message rate: 1182.07 packets/s
  Data received: 354620000000 B
  Data rate: 5.91033e+08 B/s

Intrahost Multicast 500KB Deep 100 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 100
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000

Test: SUCCESS
Settings:
  Transport type: UDP Multicast
  Buffer size: 50000000 B
  Hosts: 
    239.255.0.1:4150
  Data size: 5000 B
  Test time: 60 s
  Num vars: 100
Latency:
  Min: 290266 ns
  Avg: 490969 ns
  Max: 1247459 ns
Throughput:
  Messages received: 33329
  Message rate: 555.483 packets/s
  Data received: 1666450000000 B
  Data rate: 2.77742e+08 B/s

Intrahost Multicast 500KB Deep 1000 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 1000
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000

Test: SUCCESS
Settings:
  Transport type: UDP Multicast
  Buffer size: 50000000 B
  Hosts: 
    239.255.0.1:4150
  Data size: 500 B
  Test time: 60 s
  Num vars: 1000
Latency:
  Min: 371605 ns
  Avg: 503778 ns
  Max: 1122374 ns
Throughput:
  Messages received: 9311
  Message rate: 155.183 packets/s
  Data received: 4655500000000 B
  Data rate: 7.75917e+07 B/s

Intrahost Multicast 500KB Deep 10000 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 10000
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000

Test: SUCCESS
Settings:
  Transport type: UDP Multicast
  Buffer size: 50000000 B
  Hosts: 
    239.255.0.1:4150
  Data size: 50 B
  Test time: 60 s
  Num vars: 10000
Latency:
  Min: 1404724 ns
  Avg: 1710724 ns
  Max: 2569736 ns
Throughput:
  Messages received: 803
  Message rate: 13.3833 packets/s
  Data received: 4015000000000 B
  Data rate: 6.69167e+06 B/s

Intrahost Unicast 500KB Deep 10 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 10 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 -u 127.0.0.1:30001 -u 127.0.0.1:30000

Test: SUCCESS
Settings:
  Transport type: UDP
  Buffer size: 50000000 B
  Hosts: 
    127.0.0.1:30001
    127.0.0.1:30000
  Data size: 50000 B
  Test time: 60 s
  Num vars: 10
Latency:
  Min: 186889 ns
  Avg: 290140 ns
  Max: 692224 ns
Throughput:
  Messages received: 252177
  Message rate: 4202.95 packets/s
  Data received: 1260885000000 B
  Data rate: 2.10148e+09 B/s

Intrahost Unicast 500KB Deep 100 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 100 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 -u 127.0.0.1:30001 -u 127.0.0.1:30000

Test: SUCCESS
Settings:
  Transport type: UDP
  Buffer size: 50000000 B
  Hosts: 
    127.0.0.1:30001
    127.0.0.1:30000
  Data size: 5000 B
  Test time: 60 s
  Num vars: 100 s
Latency:
  Min: 191613 ns
  Avg: 309134 ns
  Max: 607367 ns
Throughput:
  Messages received: 111096
  Message rate: 1851.6 packets/s
  Data received: 5554800000000 B
  Data rate: 9.258e+08 B/s

Intrahost Unicast 500KB Deep 1000 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 1000 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 -u 127.0.0.1:30001 -u 127.0.0.1:30000

Subscriber received no data.
Test: FAIL.

Intrahost Unicast 500KB Deep 10000 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 10000 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 -u 127.0.0.1:30001 -u 127.0.0.1:30000

Test: SUCCESS
Settings:
  Transport type: UDP
  Buffer size: 50000000 B
  Hosts: 
    127.0.0.1:30001
    127.0.0.1:30000
  Data size: 50 B
  Test time: 60 s
  Num vars: 10000 s
Latency:
  Min: 1263935 ns
  Avg: 1349646 ns
  Max: 2012156 ns
Throughput:
  Messages received: 1737
  Message rate: 28.95 packets/s
  Data received: 8685000000000 B
  Data rate: 1.4475e+07 B/s

Intrahost ZeroMQ TCP 500KB Deep 10 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 10 --zmq tcp://127.0.0.1:30000 --zmq tcp://127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 --zmq tcp://127.0.0.1:30001 --zmq tcp://127.0.0.1:30000

Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 50000000 B
  Hosts: 
    tcp://127.0.0.1:30001
    tcp://127.0.0.1:30000
  Data size: 50000 B
  Test time: 60 s
  Num vars: 10
Latency:
  Min: 445213 ns
  Avg: 63617023 ns
  Max: 122933905 ns
Throughput:
  Messages received: 556500
  Message rate: 9275 packets/s
  Data received: 2782500000000 B
  Data rate: 4.6375e+09 B/s

Intrahost ZeroMQ TCP 500KB Deep 100 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 100 --zmq tcp://127.0.0.1:30000 --zmq tcp://127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 --zmq tcp://127.0.0.1:30001 --zmq tcp://127.0.0.1:30000

Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 50000000 B
  Hosts: 
    tcp://127.0.0.1:30001
    tcp://127.0.0.1:30000
  Data size: 5000 B
  Test time: 60 s
  Num vars: 100
Latency:
  Min: 1002983 ns
  Avg: 380304991 ns
  Max: 470933118 ns
Throughput:
  Messages received: 226875
  Message rate: 3781.25 packets/s
  Data received: 11343750000000 B
  Data rate: 1.89062e+09 B/s

Intrahost ZeroMQ TCP 500KB Deep 1000 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 1000 --zmq tcp://127.0.0.1:30000 --zmq tcp://127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 --zmq tcp://127.0.0.1:30001 --zmq tcp://127.0.0.1:30000

Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 50000000 B
  Hosts: 
    tcp://127.0.0.1:30001
    tcp://127.0.0.1:30000
  Data size: 500 B
  Test time: 60 s
  Num vars: 1000
Latency:
  Min: 1037863 ns
  Avg: 2879933181 ns
  Max: 3319101269 ns
Throughput:
  Messages received: 31688
  Message rate: 528.133 packets/s
  Data received: 15844000000000 B
  Data rate: 2.64067e+08 B/s

Intrahost ZeroMQ TCP 500KB Deep 10000 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 10000 --zmq tcp://127.0.0.1:30000 --zmq tcp://127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 --zmq tcp://127.0.0.1:30001 --zmq tcp://127.0.0.1:30000

Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 50000000 B
  Hosts: 
    tcp://127.0.0.1:30001
    tcp://127.0.0.1:30000
  Data size: 50 B
  Test time: 60 s
  Num vars: 10000
Latency:
  Min: 2035759 ns
  Avg: 13986589102 ns
  Max: 27250886271 ns
Throughput:
  Messages received: 2948
  Message rate: 49.1333 packets/s
  Data received: 14740000000000 B
  Data rate: 2.45667e+07 B/s

Intrahost ZeroMQ IPC 500KB Deep 10 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 10 --zmq ipc:///tmp/network_profiler_6_0 --zmq ipc:///tmp/network_profiler_6_1
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 --zmq ipc:///tmp/network_profiler_6_1 --zmq ipc:///tmp/network_profiler_6_0

Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 50000000 B
  Hosts: 
    ipc:///tmp/network_profiler_6_1
    ipc:///tmp/network_profiler_6_0
  Data size: 50000 B
  Test time: 60 s
  Num vars: 10
Latency:
  Min: 260138 ns
  Avg: 113840255 ns
  Max: 172763845 ns
Throughput:
  Messages received: 580223
  Message rate: 9670.38 packets/s
  Data received: 2901115000000 B
  Data rate: 4.83519e+09 B/s

Intrahost ZeroMQ IPC 500KB Deep 100 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 100 --zmq ipc:///tmp/network_profiler_6_0 --zmq ipc:///tmp/network_profiler_6_1
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 --zmq ipc:///tmp/network_profiler_6_1 --zmq ipc:///tmp/network_profiler_6_0

Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 50000000 B
  Hosts: 
    ipc:///tmp/network_profiler_6_1
    ipc:///tmp/network_profiler_6_0
  Data size: 5000 B
  Test time: 60 s
  Num vars: 100
Latency:
  Min: 2743284 ns
  Avg: 381107080 ns
  Max: 478065306 ns
Throughput:
  Messages received: 231005
  Message rate: 3850.08 packets/s
  Data received: 11550250000000 B
  Data rate: 1.92504e+09 B/s

Intrahost ZeroMQ IPC 500KB Deep 1000 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 1000 --zmq ipc:///tmp/network_profiler_6_0 --zmq ipc:///tmp/network_profiler_6_1
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 --zmq ipc:///tmp/network_profiler_6_1 --zmq ipc:///tmp/network_profiler_6_0

Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 50000000 B
  Hosts: 
    ipc:///tmp/network_profiler_6_1
    ipc:///tmp/network_profiler_6_0
  Data size: 500 B
  Test time: 60 s
  Num vars: 1000
Latency:
  Min: 1199247 ns
  Avg: 2826985827 ns
  Max: 3229973013 ns
Throughput:
  Messages received: 32322
  Message rate: 538.7 packets/s
  Data received: 16161000000000 B
  Data rate: 2.6935e+08 B/s

Intrahost ZeroMQ IPC 500KB Deep 10000 Vars

Publisher: $MADARA_ROOT/bin/network_profiler -i 0 -s 500000 -q 50000000 --num-vars 10000 --zmq ipc:///tmp/network_profiler_6_0 --zmq ipc:///tmp/network_profiler_6_1
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 --zmq ipc:///tmp/network_profiler_6_1 --zmq ipc:///tmp/network_profiler_6_0

Test: SUCCESS
Settings:
  Transport type: 0MQ
  Buffer size: 50000000 B
  Hosts: 
    ipc:///tmp/network_profiler_6_1
    ipc:///tmp/network_profiler_6_0
  Data size: 50 B
  Test time: 60 s
  Num vars: 10000
Latency:
  Min: 2328978 ns
  Avg: 13400824676 ns
  Max: 26612199414 ns
Throughput:
  Messages received: 2980
  Message rate: 49.6667 packets/s
  Data received: 14900000000000 B
  Data rate: 2.48333e+07 B/s

Intrahost Summary

  • Both UDP unicast and multicast can be used for intraprocess communication, but unicast tends to be better latency and throughput due to the copy cost of multicast as implemented by the operating system
  • There is no real comparison between multi-threaded performance and networked multi-process performance. Use multi-threading with a single knowledge base wherever possible for maximum performance. This gets even more drastic in performance difference as you cross the UDP datagram boundary (64KB)

Interhost Performance

Interhost performance focuses on the capability of the operating system and network to handle knowledge sharing between knowledge bases on two or more hosts. Interhost performance is facilitated by knowledge transports such as UDP unicast, broadcast, multicast, DDS, and ZeroMQ.

Interhost network_profiler

The $MADARA_ROOT/bin/network_profiler tool can be used for testing most supported knowledge base transports including UDP unicast, broadcast, multicast, ZeroMQ, and DDS. The tool comes with built-in help (--help or -h options) and can be run on inter-and intra-process communication between knowledge bases on multiple hosts.

To run network_profiler on two hosts for intrahost tests, open one terminal on each host and launch the network_profiler tool in each terminal window. At least one network_profiler should be id 0 (-i 0, which is the publisher and default id), and at least one network_profiler should be not zero (e.g., i 1, which is a subscriber). The publisher will publish data of a user-specified size and frequency (default is to publish as fast as possible). The subscriber will receive data and post latency and throughput information for the configured QoS. This tool is very valuable to understand performance.

Below are some example runs on an Ubuntu 16.04 Virtual Machine for interhost testing.

Interhost Multicast Performance Small

Publisher: $MADARA_ROOT/bin/network_profiler
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1

Interhost Multicast Performance Medium (64KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 64000
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1

Interhost Multicast Performance Large (500KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1

Interhost Multicast Performance Large (500KB) Deep (50MB buffer)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 -q 50000000
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000

Interhost Unicast Performance Small

Publisher: $MADARA_ROOT/bin/network_profiler -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -u 127.0.0.1:30001

Interhost Unicast Performance Medium (64KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 64000 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -u 127.0.0.1:30001

Interhost Unicast Performance Large (500KB)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -u 127.0.0.1:30001

Interhost Unicast Performance Large (500KB) Deep (50MB buffer)

Publisher: $MADARA_ROOT/bin/network_profiler -s 500000 -q 50000000 -u 127.0.0.1:30000 -u 127.0.0.1:30001
Subscriber: $MADARA_ROOT/bin/network_profiler -i 1 -q 50000000 -u 127.0.0.1:30001

Interhost Summary

  • Both UDP unicast and multicast can be used for intraprocess communication, but unicast tends to be better latency and throughput due to the copy cost of multicast as implemented by the operating system
  • There is no real comparison between multi-threaded performance and networked multi-process performance. Use multi-threading with a single knowledge base wherever possible for maximum performance. This gets even more drastic in performance difference as you cross the UDP datagram boundary (64KB)

More Information

For performance related tuning, you may want to check out the OptimizingKarl Wiki

Architecture | Knowledge Base | Networking | Containers | RCW | Threads | Optimizations | KaRL | Encryption | Checkpointing | Knowledge Performance | Logging

Add a custom footer
Add a custom sidebar
Clone this wiki locally