Skip to content

Commit

Permalink
Initial code
Browse files Browse the repository at this point in the history
Co-authored-by: Al Spinks <[email protected]>
Co-authored-by: Andrew Baranec <[email protected]>
Co-authored-by: Charles Wright <[email protected]>
Co-authored-by: Chip Kent <[email protected]>
Co-authored-by: Colin Alworth <[email protected]>
Co-authored-by: Corey Kosak <[email protected]>
Co-authored-by: Cristian Ferretti <[email protected]>
Co-authored-by: Danila Ladner <[email protected]>
Co-authored-by: Darin Petty <[email protected]>
Co-authored-by: David Godinez <[email protected]>
Co-authored-by: Davor Spasovski <[email protected]>
Co-authored-by: Don McKenzie <[email protected]>
Co-authored-by: James Nelson <[email protected]>
Co-authored-by: Jianfeng Mao <[email protected]>
Co-authored-by: Jo Janssens <[email protected]>
Co-authored-by: Matthew Runyon <[email protected]>
Co-authored-by: Mike Bender <[email protected]>
Co-authored-by: Nate Bauernfeind <[email protected]>
Co-authored-by: Pete Goddard <[email protected]>
Co-authored-by: Robb Camera <[email protected]>
Co-authored-by: Ryan Caudy <[email protected]>
Co-authored-by: Vlad Babich <[email protected]>
  • Loading branch information
1 parent 11ad079 commit c76e6e7
Show file tree
Hide file tree
Showing 7,404 changed files with 1,438,135 additions and 0 deletions.
The diff you're trying to view is too large. We only load the first 3000 changed files.
4 changes: 4 additions & 0 deletions .env
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
COMPOSE_PROJECT_NAME=core
REPO=ghcr.io/
TAG=main
PORT=10000
36 changes: 36 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
name: Bug report
about: Create a report to help us improve
labels: bug, triage

---

<!-- Please read our Code of Conduct: https://github.com/deephaven/core/blob/main/CODE_OF_CONDUCT.md -->
<!-- Please search existing issues to avoid creating duplicates. -->

**Description**

A clear and concise description of what the bug is.

**Steps to reproduce**
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'

**Expected results**

A clear and concise description of what you expected to happen.

**Actual results**

A clear and concise description of what actually happened.

**Additional details and attachments**

If applicable, add any additional screenshots, logs, or other attachments to help explain your problem.

**Versions**
- Deephaven: ...
- OS: ...
- Browser: ...
- Docker: ...
13 changes: 13 additions & 0 deletions .github/ISSUE_TEMPLATE/epic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
name: Epic
about: A large body of work that can be broken down into a number of feature requests
labels: epic, triage

---

<!-- Please read our Code of Conduct: https://github.com/deephaven/core/blob/main/CODE_OF_CONDUCT.md -->
<!-- Please search existing issues to avoid creating duplicates. -->

[Description of body of work]

<!-- After creation of the epic, create new Feature request issues and mention this epic to have them link. -->
12 changes: 12 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
name: Feature request
about: Suggest an idea for this project
labels: feature request, triage

---

<!-- Please read our Code of Conduct: https://github.com/deephaven/core/blob/main/CODE_OF_CONDUCT.md -->
<!-- Please search existing issues to avoid creating duplicates. -->

<!-- Describe the feature you'd like. -->
As a [user or stakeholder type], I want [some software feature] so that [some business value]
9 changes: 9 additions & 0 deletions .github/env/Linux/gradle.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
org.gradle.parallel=true
org.gradle.workers.max=2
org.gradle.jvmargs=-Xmx6g

# Our CI JDKs should be pre-provisioned and invoked correctly,
# we shouldn't rely on gradle for any of this logic.
org.gradle.java.installations.auto-download=false
org.gradle.java.installations.auto-detect=false
41 changes: 41 additions & 0 deletions .github/tools/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# GitHub Workflow Tools

## Nightly Benchmark Metrics and Dashboarding

Location: `.github/tools/metrics/benchmark_capture.py`

The benchmark capture tool has been created to run after the nightly benchmark job in an effort to capture result data and publish it to GCP Monitoring for historical reference. This script consumes the CSV files that the benchmark gradle jobs publish to the `tmp/logs/` folder at the root of the project during the `Nightly Benchmark` workflow execution.

### Benchmark File Format
Due to constraints from the GCP API we are limited to 10 labels per metric. This is a limitation imposed by GCP and to accomodate this we have a check to only process the first 9 fields, the `Benchmark` field is skipped as it is used as the metric's base name.

#### Baseline Fields
- Benchmark: Name of the benchmark, used as the base of the metric name.
- Score: Value used as the metric point.
- Run & Iteration are not used in the dashboarding and the mean average of each metric combination is redered as a single data point.

#### Dynamic Fields
Any fields beyond the "Benchmark", "Score", "Run" and "Iteration" are added as labels to allow for the filtering and aggregation of the various permutations for all benchmark results.

### Dashboard Creation

For each CSV file processed by the tooling a dashboard will be generated to render the metrics. Each dashboard is limited to rendering a total of 40 metrics, this is a customizable value from the commandline but has been chosen as a default in line with the upper limit from GCP for metrics per dashboard.

The dashboard title is generated from the filename basename and postfixed by the dashboard index, in the even there are greater than 40 metrics total then you will see multiple dashboards created to support this.

### GitHub Actions Integration

Location: `.github/workflows/nightly-benchmarks.yml`

The `nightly-benchmarks.yml` file includes a `benchmarks` section which can be extended to expand the scope of benchmarks run and published, for each benchmark step defined the CSV files will be consumed in the later publish-metrics step as well as be archived to GitHub artifacts by the archive step.

The following template can be used to create additional benchmark steps within the `benchmark` job definition.

```
- name: Benchmark - [BENCHMARK_NAME]
uses: burrunan/gradle-cache-action@v1
with:
job-id: benchmark
arguments: [GRADLE_COMMAND]
gradle-version: wrapper
```
43 changes: 43 additions & 0 deletions .github/tools/deephaven-gh-runner-init.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
#!/bin/bash

set -e

########################################
## GitHub Runner Installation
## Deephaven
########################################

GH_RUNNER_VERSION=${GH_RUNNER_VERSION:="2.277.1"}
GH_RUNNER_REPO_URL=${GH_RUNNER_REPO_URL:="https://github.com/deephaven/core"}
GH_RUNNER_TOKEN=${GH_RUNNER_TOKEN:?""}
GH_RUNNER_NAME=${GH_RUNNER_NAME:=$HOSTNAME}
GH_RUNNER_LABELS=${GH_RUNNER_LABELS:="gce-runner,benchmark"}
GH_RUNNER_ROOT=${GH_RUNNER_ROOT:="/github"}
GH_RUNNER_TMP=${GH_RUNNER_TMP:="/github-tmp"}

## Prepare Apt system packages
apt update
apt install --yes curl unzip docker.io

## JDK Build Prereq
apt install --yes openjdk-8-jdk-headless

## Setup runner directories
mkdir -p $GH_RUNNER_ROOT
mkdir -p $GH_RUNNER_TMP

## Pull runner binary
curl -o actions.tar.gz --location "https://github.com/actions/runner/releases/download/v${GH_RUNNER_VERSION}/actions-runner-linux-x64-${GH_RUNNER_VERSION}.tar.gz"
tar xzf actions.tar.gz --directory $GH_RUNNER_ROOT
rm -f actions.tar.gz

## Install dependencies
/github/bin/installdependencies.sh

## Configure runner
RUNNER_ALLOW_RUNASROOT=1 $GH_RUNNER_ROOT/config.sh --unattended --replace --work $GH_RUNNER_TMP --url "$GH_RUNNER_REPO_URL" --token "$GH_RUNNER_TOKEN" --labels $GH_RUNNER_LABELS

## Configure runner as service
cd $GH_RUNNER_ROOT || exit
./svc.sh install
./svc.sh start
Loading

0 comments on commit c76e6e7

Please sign in to comment.