diff --git a/docs/config.md b/docs/config.md index 803b4414..c4afbfab 100644 --- a/docs/config.md +++ b/docs/config.md @@ -143,7 +143,7 @@ default_data_file: my-experiment.data **artifact_review:** Enable [artifact evaluation](https://www.artifact-eval.org/) mode. -This mode does report less warnings and errors, but does not change +This mode reports fewer warnings and errors, but does not change how benchmarking is done. Experience shows that reviewers may misunderstand possibly chatty warnings and misinterpret them as a signs for an artifact of insufficient quality. @@ -419,17 +419,19 @@ runs: retries_after_failure: 3 ``` +--- + ## Reporting -Currently, [Codespeed] is the only supported system for continuous -performance monitoring. It is configured with the `reporting` key. +[Codespeed] and [ReBenchDB] are the currently supported system for continuous +performance monitoring. They are configured with the `reporting` key. **codespeed:** Send results to Codespeed for continuous performance tracking. The settings define the project that is configured in Codespeed, and the URL to which the results will be reported. Codespeed requires more information, -but since these details depend on the environment, other settings are passed via +but since these details depend on the environment, additional details are passed via the [command line](usage.md#continuous-performance-tracking). Example: @@ -443,6 +445,23 @@ reporting: --- +**rebenchdb:** + +Send results to a ReBenchDB server for continuous performance tracking. +The use of ReBenchDB will also require additional details that are passed via +the [command line](usage.md#continuous-performance-tracking). + +```yaml +reporting: + rebenchdb: + db_url: https://example.org/rebenchdb + repo_url: https://example.org/project.git + project_name: Some Project that Wants Good Performance + record_all: true +``` + +--- + ## Benchmark Suites Benchmark suites are named collections of benchmarks and settings that apply to diff --git a/docs/usage.md b/docs/usage.md index 15a078e2..6d467cda 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -156,26 +156,26 @@ options: #### Continuous Performance Tracking -ReBench supports [Codespeed][1] as platform for continuous performance -tracking. To report data to a Codespeed setup, the [configuration](config.md#codespeed) +ReBench supports [Codespeed][1] and [ReBenchDB][2] as platforms for continuous performance +tracking. To report data to such a setup, the [configuration](config.md#reporting) needs to have the corresponding details. -And, Codespeed needs details on the concrete execution: +Additionally, Codespeed and ReBenchDB need details on the concrete execution: ```text ---commit-id=COMMIT_ID MANDATORY: when Codespeed reporting is used, the +--commit-id COMMIT_ID MANDATORY: when Codespeed reporting is used, the commit-id has to be specified. ---environment=ENVIRONMENT MANDATORY: name the machine on which the results are +--environment ENVIRONMENT MANDATORY: name the machine on which the results are obtained. ---branch=BRANCH The branch for which the results have to be recorded, +--branch BRANCH The branch for which the results have to be recorded, i.e., to which the commit belongs. Default: HEAD ---executable=EXECUTABLE The executable name given to Codespeed. Default: The +--executable EXECUTABLE The executable name given to Codespeed. Default: The name used for the executor. ---project=PROJECT The project name given to Codespeed. Default: Value +--project PROJECT The project name given to Codespeed. Default: Value given in the config file. -I, --disable-inc-report Creates a final report at the end instead of reporting @@ -183,6 +183,11 @@ And, Codespeed needs details on the concrete execution: -R, --disable-data-reporting Override the configuration and disable any reporting to Codespeed and ReBenchDB. + +--git-repo GIT_REPO Path to the git repository that contains the source of this experiment. + This is useful when the experiment is run from a different path + without the `.git` folder. ``` -[1]: https://github.com/tobami/codespeed/ +[1]: https://github.com/tobami/codespeed +[2]: https://github.com/smarr/ReBenchDB diff --git a/rebench/rebench-schema.yml b/rebench/rebench-schema.yml index 7c6669ed..ca5aaa18 100644 --- a/rebench/rebench-schema.yml +++ b/rebench/rebench-schema.yml @@ -91,9 +91,6 @@ schema;reporting_type: project_name: type: str desc: Project name to be used by ReBenchDB. - repo_base_branch: - type: str - desc: The name of the base branch for comparisons. record_all: type: bool desc: All experiments should be stored in the ReBenchDB