Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add missing documentation for ReBenchDB #261

Merged
merged 1 commit into from
Sep 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 23 additions & 4 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ default_data_file: my-experiment.data
**artifact_review:**

Enable [artifact evaluation](https://www.artifact-eval.org/) mode.
This mode does report less warnings and errors, but does not change
This mode reports fewer warnings and errors, but does not change
how benchmarking is done. Experience shows that reviewers may misunderstand
possibly chatty warnings and misinterpret them as a signs for an artifact of
insufficient quality.
Expand Down Expand Up @@ -419,17 +419,19 @@ runs:
retries_after_failure: 3
```

---

## Reporting

Currently, [Codespeed] is the only supported system for continuous
performance monitoring. It is configured with the `reporting` key.
[Codespeed] and [ReBenchDB] are the currently supported system for continuous
performance monitoring. They are configured with the `reporting` key.

**codespeed:**

Send results to Codespeed for continuous performance tracking.
The settings define the project that is configured in Codespeed, and the
URL to which the results will be reported. Codespeed requires more information,
but since these details depend on the environment, other settings are passed via
but since these details depend on the environment, additional details are passed via
the [command line](usage.md#continuous-performance-tracking).

Example:
Expand All @@ -443,6 +445,23 @@ reporting:

---

**rebenchdb:**

Send results to a ReBenchDB server for continuous performance tracking.
The use of ReBenchDB will also require additional details that are passed via
the [command line](usage.md#continuous-performance-tracking).

```yaml
reporting:
rebenchdb:
db_url: https://example.org/rebenchdb
repo_url: https://example.org/project.git
project_name: Some Project that Wants Good Performance
record_all: true
```

---

## Benchmark Suites

Benchmark suites are named collections of benchmarks and settings that apply to
Expand Down
23 changes: 14 additions & 9 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,33 +156,38 @@ options:

#### Continuous Performance Tracking

ReBench supports [Codespeed][1] as platform for continuous performance
tracking. To report data to a Codespeed setup, the [configuration](config.md#codespeed)
ReBench supports [Codespeed][1] and [ReBenchDB][2] as platforms for continuous performance
tracking. To report data to such a setup, the [configuration](config.md#reporting)
needs to have the corresponding details.

And, Codespeed needs details on the concrete execution:
Additionally, Codespeed and ReBenchDB need details on the concrete execution:

```text
--commit-id=COMMIT_ID MANDATORY: when Codespeed reporting is used, the
--commit-id COMMIT_ID MANDATORY: when Codespeed reporting is used, the
commit-id has to be specified.

--environment=ENVIRONMENT MANDATORY: name the machine on which the results are
--environment ENVIRONMENT MANDATORY: name the machine on which the results are
obtained.

--branch=BRANCH The branch for which the results have to be recorded,
--branch BRANCH The branch for which the results have to be recorded,
i.e., to which the commit belongs. Default: HEAD

--executable=EXECUTABLE The executable name given to Codespeed. Default: The
--executable EXECUTABLE The executable name given to Codespeed. Default: The
name used for the executor.

--project=PROJECT The project name given to Codespeed. Default: Value
--project PROJECT The project name given to Codespeed. Default: Value
given in the config file.

-I, --disable-inc-report Creates a final report at the end instead of reporting
incrementally.

-R, --disable-data-reporting Override the configuration and disable any reporting
to Codespeed and ReBenchDB.

--git-repo GIT_REPO Path to the git repository that contains the source of this experiment.
This is useful when the experiment is run from a different path
without the `.git` folder.
```

[1]: https://github.com/tobami/codespeed/
[1]: https://github.com/tobami/codespeed
[2]: https://github.com/smarr/ReBenchDB
3 changes: 0 additions & 3 deletions rebench/rebench-schema.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,9 +91,6 @@ schema;reporting_type:
project_name:
type: str
desc: Project name to be used by ReBenchDB.
repo_base_branch:
type: str
desc: The name of the base branch for comparisons.
record_all:
type: bool
desc: All experiments should be stored in the ReBenchDB
Expand Down