Skip to content

Commit

Permalink
Smallfile cr params (cloud-bulldozer#652)
Browse files Browse the repository at this point in the history 
* add auto-pause related params, allow all params from CR

* output smallfile per-thread logs in debug mode

* remove unusable smallfile params, add missing

* clarify smallfile example, point to documentation

* add spaces to improve consistency, readability

remove unsupported parameter

* improve consistency, readability, and robustness

do not assume clustername is defined in CR
use redis_host with publisher to clarify meaning
add spaces inside double squiggly brackets for readability

* test auto-pause param in CI

* redis_host not defined to jinja2

* update and link to github smallfile repo doc

do not exhaustively track all parameters here,
link to where they are defined,
and indicate which ones NOT to specify for benchmark-operator
also give a more realistic example log

* oops on doc

* more concise defaulting, use single-quote inside double-quote

* pause and auto-pause are mutually exclusive
  • Loading branch information
@bengland2
bengland2 authored Sep 8, 2021
1 parent 8a75514 commit 2f834bd
Show file tree
Hide file tree
Showing 5 changed files with 119 additions and 181 deletions.
12 changes: 9 additions & 3 deletions config/samples/smallfile/cr.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,23 @@ metadata:
namespace: benchmark-operator
spec:
test_user: homer_simpson
clustername: aws-2009-09-10
clustername: nuclear-power-plant
elasticsearch:
url: http://my.elasticsearch.server:9200
workload:
name: smallfile
args:
#drop_cache_kernel: true
#drop_cache_rook_ceph: true
#storageclass: my-kubernetes-storageclass
storagesize: 2Gi
clients: 1
samples: 1
operation: ["create", "read"]
threads: 5
file_size: 64
files: 100
storageclass: rook-ceph-block
storagesize: 2Gi
# more parameters available, see smallfile documentation
# at https://github.com/distributed-system-analysis/smallfile/blob/master/README.md
# subset of parameters supported for benchmark-operator are in
# roles/smallfile/templates/smallfilejob.other_parameters.yaml.j2
198 changes: 60 additions & 138 deletions docs/smallfile.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,104 +5,28 @@
## Running Smallfile Benchmark using Ripsaw
Once the operator has been installed following the instructions, one needs to modify the clients parameter(which is currently set to 0), to value greater than 0 in [cr.yaml](../config/samples/smallfile/cr.yaml) to run default "create" the test. Also, in addtion to that, smallfile operator is completely dependent on storageclass and storagesize. Please make sure to double check the parameters in CRD file.

```yaml
apiVersion: ripsaw.cloudbulldozer.io/v1alpha1
kind: Benchmark
metadata:
name: example-benchmark
namespace: benchmark-operator
spec:
workload:
name: smallfile
args:
clients: 1
operation: ["create"]
threads: 5
file_size: 64
files: 100
storageclass: rook-ceph-block # Provide if PV is needed
storagesize: 5Gi # Provide if PV is needed
```
Smallfile operator also gives the leverage to run multiple test operations in a user-defined sequence. Like in the [Custom Resource Definition file](../config/samples/smallfile/cr.yaml), the series of operation can be mentioned as:
Smallfile operator also gives the leverage to run multiple test operations in a user-defined sequence. Like in the [Custom Resource Definition file](../config/samples/smallfile/cr.yaml), the series of operations can be specified as a list.

```yaml
apiVersion: ripsaw.cloudbulldozer.io/v1alpha1
kind: Benchmark
metadata:
name: example-benchmark
namespace: benchmark-operator
spec:
workload:
name: smallfile
args:
clients: 1
operation: ["create","read","append", "delete"]
threads: 5
file_size: 64
files: 100
storageclass: rook-ceph-block # Provide if PV is needed
storagesize: 5Gi # Provide if PV is needed
```
NOTE: While running the sequence of tests using smallfile workload, please make sure that the initial operation must be create, and the "cleanup" operation should come in termination, else smallfile might produce error due to meaningless sequence of tests. For example:

NOTE: While running the sequence of tests using smallfile workload, please make sure that the initial operation must be create, and the "delete" operation should come in termination, else smallfile might produce error due to meaningless sequence of tests. For example:
```bash
operation: ["read","append","create", "delete"]

operation: ["read","append","create", "delete","cleanup"]
#This will be meaningless sequence of test as trying to read something, which has not been created yet.The same logic applies for the append test as well. Hence, smallfile will produce error.
```

## Adding More options in smallfile tests

Smallfile also comes with a variety of configurable options for running tests, following are the parameters that can be added to CR file:

* **response_times** – if Y then save response time for each file operation in a
rsptimes\*csv file in the shared network directory. Record format is
operation-type, start-time, response-time. The operation type is included so
that you can run different workloads at the same time and easily merge the
data from these runs. The start-time field is the time that the file
operation started, down to microsecond resolution. The response time field is
the file operation duration down to microsecond resolution.
* **file_size_distribution** – only supported value today is exponential.
* **record_size** -- record size in KB, how much data is transferred in a single
read or write system call. If 0 then it is set to the minimum of the file
size and 1-MiB record size limit.
* **files_per_dir** -- maximum number of files contained in any one directory.
* **dirs_per_dir** -- maximum number of subdirectories contained in any one
directory.
* **hash_into_dirs** – if Y then assign next file to a directory using a hash
function, otherwise assign next –files-per-dir files to next directory.
* **same_dir** -- if Y then threads will share a single directory.
* **network_sync_dir** – don't need to specify unless you run a multi-host test
and the –top parameter points to a non-shared directory (see discussion
below). Default: network_shared subdirectory under –top dir.
* **permute_host_dirs** – if Y then have each host process a different
subdirectory tree than it otherwise would (see below for directory tree
structure).
* **xattr_size** -- size of extended attribute value in bytes (names begin with
'user.smallfile-')
* **xattr_count** -- number of extended attributes per file
* **prefix** -- a string prefix to prepend to files (so they don't collide with
previous runs for example)
* **suffix** -- a string suffix to append to files (so they don't collide with
previous runs for example)
* **incompressible** – if Y then generate a pure-random file that
will not be compressible (useful for tests where intermediate network or file
copy utility attempts to compress data
* **record_ctime_size** -- if Y then label each created file with an
xattr containing a time of creation and a file size. This will be used by
–await-create operation to compute performance of asynchonous file
replication/copy.
* **finish** -- if Y, thread will complete all requested file operations even if
measurement has finished.
* **stonewall** -- if Y then thread will measure throughput as soon as it detects
that another thread has finished.
* **verify_read** – if Y then smallfile will verify read data is correct.
* **remote_pgm_dir** – don't need to specify this unless the smallfile software
lives in a different directory on the target hosts and the test-driver host.
* **pause** -- integer (microseconds) each thread will wait before starting next
file.
* **runtime_class** - If this is set, the benchmark-operator will apply the runtime_class to the podSpec runtimeClassName.
Smallfile also comes with a variety of configurable options that can be added by the user to the CR for running tests, documented [at its github site here](https://github.com/distributed-system-analysis/smallfile#readme) . To obtain the YAML parameter name, remove the 2 preceding dashes and convert remaining dashes to underscores and append a colon character. For example, **--file-size** becomes **file_size:** . Parameters that **are not** usable in the smallfile CR include:

* --yaml-input-file - used by the benchmark
* --operation - you specify this in the CR in the operation list
* --top - you do not specify this, Kubernetes points smallfile to the PV mountpoint
* --response-times - used by the benchmark
* --output-json - used by the benchmark
* --network-sync-dir - not applicable for pods
* --permute-host-dirs - not applicable for pods
* --remote-pgm-dir - not applicable for pods

Once done creating/editing the resource file, one can run it by:

Expand All @@ -117,57 +41,55 @@ $ kubectl get pods
NAME READY STATUS RESTARTS AGE
benchmark-operator-7c6bc98b8c-2j5x5 2/2 Running 0 47s
example-benchmark-smallfile-client-1-benchmark-hwj4b 1/1 Running 0 33s
example-benchmark-smallfile-publisher...
```

To see the output of the run one has to run `kubectl logs <client>`. This is shown below:
To see the output of the run one has to run `kubectl logs <client>`. This looks *approximately* like:
```bash
$ kubectl logs example-benchmark-smallfile-client-1-benchmark-hwj4b
create
RUnning_OPERATION_create

top: /mnt/pvc/smallfile_test_data
threads: 5
file-size: 64
files: 100
version : 3.1
hosts in test : None
launch by daemon : False
top test directory(s) : ['/mnt/pvc/smallfile_test_data']
operation : create
files/thread : 100
threads : 5
record size (KB, 0 = maximum) : 0
file size (KB) : 64
file size distribution : fixed
files per dir : 100
dirs per dir : 10
threads share directories? : N
filename prefix :
filename suffix :
hash file number into dir.? : N
fsync after modify? : N
pause between files (microsec) : 0
minimum directories per sec : 50
finish all requests? : Y
stonewall? : Y
measure response times? : N
verify read? : Y
verbose? : False
log to stderr? : False
ext.attr.size : 0
ext.attr.count : 0
host = example-benchmark-smallfile-client-1-benchmark-hwj4b,thr = 00,elapsed = 0.015719,files = 100,records = 100,status = ok
host = example-benchmark-smallfile-client-1-benchmark-hwj4b,thr = 01,elapsed = 0.020679,files = 100,records = 100,status = ok
host = example-benchmark-smallfile-client-1-benchmark-hwj4b,thr = 02,elapsed = 0.028791,files = 100,records = 100,status = ok
host = example-benchmark-smallfile-client-1-benchmark-hwj4b,thr = 03,elapsed = 0.029367,files = 100,records = 100,status = ok
host = example-benchmark-smallfile-client-1-benchmark-hwj4b,thr = 04,elapsed = 0.029750,files = 100,records = 100,status = ok
total threads = 5
total files = 500
total IOPS = 500
total data = 0.031 GiB
100.00% of requested files processed, minimum is 70.00
elapsed time = 0.030
files/sec = 16806.661271
IOPS = 16806.661271
MiB/sec = 1050.41633
$ kubectl logs example-benchmark-smallfile-client-1-benchmark-hwj4b
Waiting For all Smallfile Pods to get ready ...
Executing Smallfile...
2021-08-24T20:26:32Z - INFO - MainProcess - trigger_smallfile: running:smallfile_cli.py --operation create --top /mnt/pvc/smallfile_test_data --output-json /var/tmp/RESULTS/1/create.json --response-times Y --yaml-input-file /tmp/smallfile/smallfilejob
2021-08-24T20:26:32Z - INFO - MainProcess - trigger_smallfile: from current directory /opt/snafu
version : 3.2
hosts in test : None
launch by daemon : False
top test directory(s) : ['/mnt/pvc/smallfile_test_data']
operation : create
files/thread : 10000
threads : 1
record size (KB, 0 = maximum) : 0
file size (KB) : 4
file size distribution : fixed
files per dir : 100
dirs per dir : 10
threads share directories? : N
filename prefix :
filename suffix :
hash file number into dir.? : N
fsync after modify? : N
pause between files (microsec) : 0
auto-pause? : N
delay after cleanup per file (microsec) : 0
minimum directories per sec : 50
total hosts : 30
finish all requests? : Y
stonewall? : Y
measure response times? : Y
verify read? : Y
verbose? : False
log to stderr? : False
host = smallfile-client-1-benchmark-84ad212e-9h454,thr = 00,elapsed = 0.592771,files = 10000,records = 10000,status = ok
total threads = 1
total files = 10000
total IOPS = 16869
total data = 0.038 GiB
100.00% of requested files processed, warning threshold is 70.00
elapsed time = 0.593
files/sec = 16869.919582
IOPS = 16869.919582
MiB/sec = 65.898123
2021-08-24T20:26:37Z - INFO - MainProcess - trigger_smallfile: completed sample 1 for operation create , results in /var/tmp/RESULTS/1/create.json
...
```
62 changes: 34 additions & 28 deletions roles/smallfile/templates/smallfilejob.other_parameters.yaml.j2
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@


top: {{ smallfile_path }}/smallfile_test_data

{% if workload_args.auto_pause is defined %}
auto-pause: {{ workload_args.auto_pause }}
{% endif %}

{% if workload_args.clients is defined %}
total-hosts: {{ workload_args.clients }}
{% endif %}

{% if workload_args.threads is defined %}
threads: {{ workload_args.threads }}
Expand All @@ -15,81 +21,81 @@ files: {{ workload_args.files }}
{% endif %}

{% if workload_args.fsync is defined %}
fsync: y
fsync: {{ workload_args.fsync }}
{% endif %}

{% if workload_args.response_times is defined %}
response-times: true
{% endif %}
# response-times is always set to Y so that
# benchmark-wrapper can collect response time data

{% if workload_args.network_sync_dir is defined %}
network-sync-dir: {{workload_args.network_sync_dir}}
{% endif %}
# network-sync-dir is never used with benchmark-operator
# since it uses redis to synchronize pods and elastic to return data

{% if workload_args.files_per_dir is defined %}
files-per-dir: {{workload_args.files_per_dir}}
files-per-dir: {{ workload_args.files_per_dir }}
{% endif %}

{% if workload_args.dirs_per_dir is defined %}
dirs-per-dir: {{workload_args.dirs_per_dir}}
dirs-per-dir: {{ workload_args.dirs_per_dir }}
{% endif %}

{% if workload_args.record_size is defined %}
record-size: {{workload_args.record_size}}
record-size: {{ workload_args.record_size }}
{% endif %}

{% if workload_args.file_size_distribution is defined %}
file-size-distribution: {{workload_args.file_size_distribution}}
file-size-distribution: {{ workload_args.file_size_distribution }}
{% endif %}

{% if workload_args.xattr_size is defined %}
xattr-size: {{workload_args.xattr_size}}
xattr-size: {{ workload_args.xattr_size }}
{% endif %}

{% if workload_args.xattr_count is defined %}
xattr-count: {{workload_args.xattr_count}}
xattr-count: {{ workload_args.xattr_count }}
{% endif %}

{% if workload_args.pause is defined %}
pause: {{workload_args.pause}}
pause: {{ workload_args.pause }}
{% endif %}

{% if workload_args.stonewall is defined %}
stonewall: {{workload_args.stonewall}}
stonewall: {{ workload_args.stonewall }}
{% endif %}

{% if workload_args.finish is defined %}
finish: {{workload_args.finish}}
finish: {{ workload_args.finish }}
{% endif %}

{% if workload_args.prefix is defined %}
prefix: {{workload_args.prefix}}
prefix: {{ workload_args.prefix }}
{% endif %}

{% if workload_args.suffix is defined %}
suffix: {{ workload_args.suffix }}
{% endif %}

{% if workload_args.hash_into_dirs is defined %}
hash-into-dirs: {{workload_args.hash_into_dirs}}
hash-into-dirs: {{ workload_args.hash_into_dirs }}
{% endif %}

{% if workload_args.same_dir is defined %}
same-dir: {{workload_args.same_dir}}
same-dir: {{ workload_args.same_dir }}
{% endif %}

{% if workload_args.verbose is defined %}
verbose: {{workload_args.verbose}}
verbose: {{ workload_args.verbose }}
{% endif %}

{% if workload_args.permute_host_dirs is defined %}
permute-host-dirs: true
{% endif %}
# permute-host-dirs is not applicable to K8S pods

{% if workload_args.record_ctime_size is defined %}
record-ctime-size: true
record-ctime-size: {{ workload_args.record_ctime_size }}
{% endif %}

{% if workload_args.verify_read is defined %}
verify-read: true
verify-read: {{ workload_args.verify_read }}
{% endif %}

{% if workload_args.incompressible is defined %}
incompressible: true
incompressible: {{ workload_args.incompressible }}
{% endif %}
Loading

0 comments on commit 2f834bd

Please sign in to comment.