From cc2ff54d77abf998690630ceded9861cffcc6f98 Mon Sep 17 00:00:00 2001 From: georgi-l95 Date: Tue, 29 Oct 2024 15:46:47 +0200 Subject: [PATCH 01/11] feat: reorder readme.md Signed-off-by: georgi-l95 --- README.md | 67 ++++++++++++++++--------------------------- docs/README.md | 26 +++++++++++++++++ docs/configuration.md | 19 ++++++++++++ 3 files changed, 70 insertions(+), 42 deletions(-) create mode 100644 docs/README.md create mode 100644 docs/configuration.md diff --git a/README.md b/README.md index aaa3e7b18..3e37272dc 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,25 @@ -# hedera-block-node -The block node is a new and unique node designed to consume the blocks streams, maintain state and expose additional targeted value adding APIs to the Hedera community. -More details to come as this is work in progress +[![Build Application](https://github.com/hashgraph/hedera-block-node/actions/workflows/build-application.yaml/badge.svg?branch=main)](https://github.com/hashgraph/hedera-block-node/actions/workflows/build-application.yaml) +[![E2E Test Suites](https://github.com/hashgraph/hedera-block-node/actions/workflows/e2e-tests.yaml/badge.svg?branch=main)](https://github.com/hashgraph/hedera-block-node/actions/workflows/e2e-tests.yaml) +[![codecov](https://codecov.io/github/hashgraph/hedera-block-node/graph/badge.svg?token=OF6T6E8V7U)](https://codecov.io/github/hashgraph/hedera-block-node) + +[![Latest Version](https://img.shields.io/github/v/tag/hashgraph/hedera-block-node?sort=semver&label=version)](README.md) +[![Made With](https://img.shields.io/badge/made_with-java-blue)](https://github.com/hashgraph/hedera-block-node/) +[![Development Branch](https://img.shields.io/badge/docs-quickstart-green.svg)](docs/gradle-quickstart.md) +[![License](https://img.shields.io/badge/license-apache2-blue.svg)](LICENSE) + +# Hedera Block Node +Implementation of the Hedera Block Node, which is responsible for consuming the block streams, maintaining state and exposing additional targeted value adding APIs to the Hedera community. + +## Overview of child modules + +- `server`: implementation of the block node, which contains the main application and all the necessary code to run the block node. +- `simulator`: A simulator for the block node, which can be used to test the block node in a local environment. +- `common`: Module responsible for holding common literals, utilities and types used by the other modules. +- `suites`: A set of e2e tests that can be used to verify the correctness of the block node. + +## Getting Started + +Refer to the [Quickstart Guide](docs/README.md) for how to work with this project. ## Support @@ -20,46 +39,10 @@ This project is governed by the participating, you are expected to uphold this code of conduct. Please report unacceptable behavior to [oss@hedera.com](mailto:oss@hedera.com). -## License - -[Apache License 2.0](LICENSE) - # 🔐 Security -Please do not file a public ticket mentioning the vulnerability. Refer to the security policy defined in the [SECURITY.md](https://github.com/hashgraph/hedera-sourcify/blob/main/SECURITY.md). - ---- - -# Usage - -## Configuration - -| Environment Variable | Description | Default Value | -|-----------------------------------|---------------------------------------------------------------------------------------------------------------|---------------| -| PERSISTENCE_STORAGE_ROOT_PATH | The root path for the storage, if not provided will attempt to create a `data` on the working dir of the app. | | -| CONSUMER_TIMEOUT_THRESHOLD_MILLIS | Time to wait for subscribers before disconnecting in milliseconds | 1500 | -| SERVICE_DELAY_MILLIS | Service shutdown delay in milliseconds | 500 | -| MEDIATOR_RING_BUFFER_SIZE | Size of the ring buffer used by the mediator (must be a power of 2) | 67108864 | -| NOTIFIER_RING_BUFFER_SIZE | Size of the ring buffer used by the notifier (must be a power of 2) | 2048 | - -# Starting locally: -```bash -./gradlew run -``` - -In debug mode, you can attach a debugger to the port 5005. -```bash -./gradlew run --debug-jvm -``` - -Also you can run on docker locally: -```bash -./gradlew startDockerContainer -``` - -# Running Tests -1) ./gradlew build +Please do not file a public ticket mentioning the vulnerability. Refer to the security policy defined in the [SECURITY.md](https://github.com/hashgraph/hedera-block-node/blob/main/SECURITY.md). +## License -### Code Coverage -[![codecov](https://codecov.io/github/hashgraph/hedera-block-node/graph/badge.svg?token=OF6T6E8V7U)](https://codecov.io/github/hashgraph/hedera-block-node) +[Apache License 2.0](LICENSE) diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..026b57f19 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,26 @@ +# Quickstart Guide + +This guide will help you get started with the Hedera Block Node. +## Configuration + +Refer to the [Configuration](docs/configuration.md) for more information on the configuration options. + +## Starting locally: +```bash +./gradlew run +``` + +In debug mode, you can attach a debugger to the port 5005. +```bash +./gradlew run --debug-jvm +``` + +Also you can run on docker locally: +```bash +./gradlew startDockerContainer +``` + +## Running Tests +```bash +./gradlew build +``` diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 000000000..f35a24fbe --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,19 @@ +# Configuration + +The components of the Hedera Block Node all support loading configuration via the +environment. + +## Default Values + +The default configuration allows users to quickly get up and running without having to configure anything. This provides +ease of use at the trade-off of some insecure default configuration. Most configuration settings have appropriate +defaults and can be left unchanged. It is recommended to browse the properties below and adjust to your needs. + + +| Environment Variable | Description | Default Value | +|-----------------------------------|---------------------------------------------------------------------------------------------------------------|---------------| +| PERSISTENCE_STORAGE_ROOT_PATH | The root path for the storage, if not provided will attempt to create a `data` on the working dir of the app. | | +| CONSUMER_TIMEOUT_THRESHOLD_MILLIS | Time to wait for subscribers before disconnecting in milliseconds | 1500 | +| SERVICE_DELAY_MILLIS | Service shutdown delay in milliseconds | 500 | +| MEDIATOR_RING_BUFFER_SIZE | Size of the ring buffer used by the mediator (must be a power of 2) | 67108864 | +| NOTIFIER_RING_BUFFER_SIZE | Size of the ring buffer used by the notifier (must be a power of 2) | 2048 | From 753c2a4bbaf460263277f6efbf633bb2eb8dec8e Mon Sep 17 00:00:00 2001 From: georgi-l95 Date: Thu, 31 Oct 2024 16:26:09 +0200 Subject: [PATCH 02/11] reorg structure Signed-off-by: georgi-l95 --- README.md | 21 ++++++++++--------- docs/README.md | 31 ++++++++++------------------ server/README.md | 17 ++++++++++++++++ server/docs/configuration.md | 19 +++++++++++++++++ simulator/README.md | 35 +------------------------------- simulator/docs/configuration.md | 36 +++++++++++++++++++++++++++++++++ 6 files changed, 96 insertions(+), 63 deletions(-) create mode 100644 server/README.md create mode 100644 server/docs/configuration.md create mode 100644 simulator/docs/configuration.md diff --git a/README.md b/README.md index 3e37272dc..15a581195 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,3 @@ -[![Build Application](https://github.com/hashgraph/hedera-block-node/actions/workflows/build-application.yaml/badge.svg?branch=main)](https://github.com/hashgraph/hedera-block-node/actions/workflows/build-application.yaml) -[![E2E Test Suites](https://github.com/hashgraph/hedera-block-node/actions/workflows/e2e-tests.yaml/badge.svg?branch=main)](https://github.com/hashgraph/hedera-block-node/actions/workflows/e2e-tests.yaml) -[![codecov](https://codecov.io/github/hashgraph/hedera-block-node/graph/badge.svg?token=OF6T6E8V7U)](https://codecov.io/github/hashgraph/hedera-block-node) - -[![Latest Version](https://img.shields.io/github/v/tag/hashgraph/hedera-block-node?sort=semver&label=version)](README.md) -[![Made With](https://img.shields.io/badge/made_with-java-blue)](https://github.com/hashgraph/hedera-block-node/) -[![Development Branch](https://img.shields.io/badge/docs-quickstart-green.svg)](docs/gradle-quickstart.md) -[![License](https://img.shields.io/badge/license-apache2-blue.svg)](LICENSE) # Hedera Block Node Implementation of the Hedera Block Node, which is responsible for consuming the block streams, maintaining state and exposing additional targeted value adding APIs to the Hedera community. @@ -16,10 +8,21 @@ Implementation of the Hedera Block Node, which is responsible for consuming the - `simulator`: A simulator for the block node, which can be used to test the block node in a local environment. - `common`: Module responsible for holding common literals, utilities and types used by the other modules. - `suites`: A set of e2e tests that can be used to verify the correctness of the block node. +- `tools`: A set of command line tools for working with block stream files. + +## Project Links +[![Build Application](https://github.com/hashgraph/hedera-block-node/actions/workflows/build-application.yaml/badge.svg?branch=main)](https://github.com/hashgraph/hedera-block-node/actions/workflows/build-application.yaml) +[![E2E Test Suites](https://github.com/hashgraph/hedera-block-node/actions/workflows/e2e-tests.yaml/badge.svg?branch=main)](https://github.com/hashgraph/hedera-block-node/actions/workflows/e2e-tests.yaml) +[![codecov](https://codecov.io/github/hashgraph/hedera-block-node/graph/badge.svg?token=OF6T6E8V7U)](https://codecov.io/github/hashgraph/hedera-block-node) + +[![Latest Version](https://img.shields.io/github/v/tag/hashgraph/hedera-block-node?sort=semver&label=version)](README.md) +[![Made With](https://img.shields.io/badge/made_with-java-blue)](https://github.com/hashgraph/hedera-block-node/) +[![Development Branch](https://img.shields.io/badge/docs-quickstart-green.svg)](docs/gradle-quickstart.md) +[![License](https://img.shields.io/badge/license-apache2-blue.svg)](LICENSE) ## Getting Started -Refer to the [Quickstart Guide](docs/README.md) for how to work with this project. +Refer to the [Hedera Block Node Documentation Overview](docs/README.md) for more information about the project, design and guides. ## Support diff --git a/docs/README.md b/docs/README.md index 026b57f19..26046cf65 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,26 +1,17 @@ -# Quickstart Guide +# Hedera Block Node Documentation Overview -This guide will help you get started with the Hedera Block Node. -## Configuration +This document will outline the structure and content of Block Node documentation. -Refer to the [Configuration](docs/configuration.md) for more information on the configuration options. +## Table of Contents -## Starting locally: -```bash -./gradlew run -``` +1. Introduction +2. Concepts -In debug mode, you can attach a debugger to the port 5005. -```bash -./gradlew run --debug-jvm -``` +## Introduction -Also you can run on docker locally: -```bash -./gradlew startDockerContainer -``` +- [Hedera Block Node](../server/README.md): Provides an introduction to Hedera Block Node, its purpose and how it can be used. +- [Simulator](../simulator/README.md): Offers instructions on how the Simulator can be utilized. -## Running Tests -```bash -./gradlew build -``` +## Concepts + +- [Hedera Block Node Protocol](Protocol/README.md): Overview of the Hedera Block Node Protocol diff --git a/server/README.md b/server/README.md new file mode 100644 index 000000000..9517d1d20 --- /dev/null +++ b/server/README.md @@ -0,0 +1,17 @@ +# Block Node Application + +## Overview + +The Block Stream Application is designed handle the output of a Hedera Node, which would be in form of Block Stream. +By handling we can understand verifying, storing and applying needed state changes. +It uses various configuration sources and dependency injection to manage its components. + +## Prerequisites + +- Java 21 +- Gradle +- IntelliJ IDEA (recommended for development) + +## Configuration + +Refer to the [Configuration](docs/configuration.md) for configuration options. diff --git a/server/docs/configuration.md b/server/docs/configuration.md new file mode 100644 index 000000000..f35a24fbe --- /dev/null +++ b/server/docs/configuration.md @@ -0,0 +1,19 @@ +# Configuration + +The components of the Hedera Block Node all support loading configuration via the +environment. + +## Default Values + +The default configuration allows users to quickly get up and running without having to configure anything. This provides +ease of use at the trade-off of some insecure default configuration. Most configuration settings have appropriate +defaults and can be left unchanged. It is recommended to browse the properties below and adjust to your needs. + + +| Environment Variable | Description | Default Value | +|-----------------------------------|---------------------------------------------------------------------------------------------------------------|---------------| +| PERSISTENCE_STORAGE_ROOT_PATH | The root path for the storage, if not provided will attempt to create a `data` on the working dir of the app. | | +| CONSUMER_TIMEOUT_THRESHOLD_MILLIS | Time to wait for subscribers before disconnecting in milliseconds | 1500 | +| SERVICE_DELAY_MILLIS | Service shutdown delay in milliseconds | 500 | +| MEDIATOR_RING_BUFFER_SIZE | Size of the ring buffer used by the mediator (must be a power of 2) | 67108864 | +| NOTIFIER_RING_BUFFER_SIZE | Size of the ring buffer used by the notifier (must be a power of 2) | 2048 | diff --git a/simulator/README.md b/simulator/README.md index 837871a15..f41f63c94 100644 --- a/simulator/README.md +++ b/simulator/README.md @@ -31,40 +31,7 @@ The BlockStreamSimulatorApp consumes other services that are injected using Dagg ## Configuration -There are 2 configuration sets: -1. BlockStreamConfig: contains the configuration for the Block Stream Simulator logic and the generation module. -2. GrpcConfig: contains the configuration for the gRPC communication with the Block-Node. - -### BlockStreamConfig -Uses the prefix `blockStream` so all properties should start with `blockStream.` - -| Key | Description | Default Value | -|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------| -| `delayBetweenBlockItems` | The delay between each block item in nanoseconds, only applicable when streamingMode=CONSTANT_RATE | `1_500_000` | -| `maxBlockItemsToStream` | exit condition for the simulator and the circular implementations such as `BlockAsDir` or `BlockAsFile` implementations | `10_000` | -| `streamingMode` | can either be `CONSTANT_RATE` or `MILLIS_PER_BLOCK` | `CONSTANT_RATE` | -| `millisecondsPerBlock` | if streamingMode is `MILLIS_PER_BLOCK` this will be the time to wait between blocks in milliseconds | `1_000` | -| `blockItemsBatchSize` | the number of block items to send in a single batch, however if a block has less block items, it will send all the items in a block | `1_000` | - -### GeneratorConfig -Uses the prefix `generator` so all properties should start with `generator.` -| Key | Description | Default Value | -|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------| -| `generationMode` | The desired generation Mode to use, it can only be `DIR` or `AD_HOC` | `DIR` | -| `folderRootPath` | If the generationMode is DIR this will be used as the source of the recording to stream to the Block-Node | `` | -| `managerImplementation` | The desired implementation of the BlockStreamManager to use, it can only be `BlockAsDirBlockStreamManager`, `BlockAsFileBlockStreamManager` or `BlockAsFileLargeDataSets` | `BlockAsFileBlockStreamManager` | -| `paddedLength` | on the `BlockAsFileLargeDataSets` implementation, the length of the padded left zeroes `000001.blk.gz` | 36 | -| `fileExtension` | on the `BlockAsFileLargeDataSets` implementation, the extension of the files to be streamed | `.blk.gz` | - - - -### GrpcConfig -Uses the prefix `grpc` so all properties should start with `grpc.` - -| Key | Description | Default Value | -|-----------------|----------------------------|---------------| -| `serverAddress` | The host of the Block-Node | `localhost` | -| `port` | The port of the Block-Node | `8080` | +Refer to the [Configuration](docs/configuration.md) for configuration options. ## Building the Project diff --git a/simulator/docs/configuration.md b/simulator/docs/configuration.md new file mode 100644 index 000000000..36bc2ea97 --- /dev/null +++ b/simulator/docs/configuration.md @@ -0,0 +1,36 @@ +## Configuration + +There are 2 configuration sets: +1. BlockStreamConfig: contains the configuration for the Block Stream Simulator logic and the generation module. +2. GrpcConfig: contains the configuration for the gRPC communication with the Block-Node. + +### BlockStreamConfig +Uses the prefix `blockStream` so all properties should start with `blockStream.` + +| Key | Description | Default Value | +|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------| +| `delayBetweenBlockItems` | The delay between each block item in nanoseconds, only applicable when streamingMode=CONSTANT_RATE | `1_500_000` | +| `maxBlockItemsToStream` | exit condition for the simulator and the circular implementations such as `BlockAsDir` or `BlockAsFile` implementations | `10_000` | +| `streamingMode` | can either be `CONSTANT_RATE` or `MILLIS_PER_BLOCK` | `CONSTANT_RATE` | +| `millisecondsPerBlock` | if streamingMode is `MILLIS_PER_BLOCK` this will be the time to wait between blocks in milliseconds | `1_000` | +| `blockItemsBatchSize` | the number of block items to send in a single batch, however if a block has less block items, it will send all the items in a block | `1_000` | + +### GeneratorConfig +Uses the prefix `generator` so all properties should start with `generator.` +| Key | Description | Default Value | +|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------| +| `generationMode` | The desired generation Mode to use, it can only be `DIR` or `AD_HOC` | `DIR` | +| `folderRootPath` | If the generationMode is DIR this will be used as the source of the recording to stream to the Block-Node | `` | +| `managerImplementation` | The desired implementation of the BlockStreamManager to use, it can only be `BlockAsDirBlockStreamManager`, `BlockAsFileBlockStreamManager` or `BlockAsFileLargeDataSets` | `BlockAsFileBlockStreamManager` | +| `paddedLength` | on the `BlockAsFileLargeDataSets` implementation, the length of the padded left zeroes `000001.blk.gz` | 36 | +| `fileExtension` | on the `BlockAsFileLargeDataSets` implementation, the extension of the files to be streamed | `.blk.gz` | + + + +### GrpcConfig +Uses the prefix `grpc` so all properties should start with `grpc.` + +| Key | Description | Default Value | +|-----------------|----------------------------|---------------| +| `serverAddress` | The host of the Block-Node | `localhost` | +| `port` | The port of the Block-Node | `8080` | \ No newline at end of file From 617572b66ec5eb831aaf50e827bcb0f9b24c4f7a Mon Sep 17 00:00:00 2001 From: Atanas Atanasov Date: Fri, 1 Nov 2024 11:17:44 +0200 Subject: [PATCH 03/11] formatting and adding tocs for root project and server project, need to finish for others Signed-off-by: Atanas Atanasov --- LICENSE | 1 - README.md | 16 +++++++- docs/Protocol/README.md | 6 +-- docs/Protocol/publishBlockStream.md | 9 ++++ docs/README.md | 9 +++- docs/configuration.md | 19 --------- docs/release.md | 28 +++++++++---- server/README.md | 24 +++++++++++ server/docs/configuration.md | 2 +- .../bidi-producer-consumers-streaming.md | 41 +++++++++++++------ server/docs/design/block-persistence.md | 18 ++++++-- server/docs/metrics.md | 16 ++++++-- 12 files changed, 133 insertions(+), 56 deletions(-) delete mode 100644 docs/configuration.md diff --git a/LICENSE b/LICENSE index 57bc88a15..261eeb9e9 100644 --- a/LICENSE +++ b/LICENSE @@ -199,4 +199,3 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. - diff --git a/README.md b/README.md index 15a581195..6a4311c43 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,18 @@ - # Hedera Block Node + Implementation of the Hedera Block Node, which is responsible for consuming the block streams, maintaining state and exposing additional targeted value adding APIs to the Hedera community. +## Table of Contents + +1. [Overview of child modules](#overview-of-child-modules) +1. [Project Links](#project-links) +1. [Getting Started](#getting-started) +1. [Support](#support) +1. [Contributing](#contributing) +1. [Code of Conduct](#code-of-conduct) +1. [Security](#-security) +1. [License](#license) + ## Overview of child modules - `server`: implementation of the block node, which contains the main application and all the necessary code to run the block node. @@ -11,6 +22,7 @@ Implementation of the Hedera Block Node, which is responsible for consuming the - `tools`: A set of command line tools for working with block stream files. ## Project Links + [![Build Application](https://github.com/hashgraph/hedera-block-node/actions/workflows/build-application.yaml/badge.svg?branch=main)](https://github.com/hashgraph/hedera-block-node/actions/workflows/build-application.yaml) [![E2E Test Suites](https://github.com/hashgraph/hedera-block-node/actions/workflows/e2e-tests.yaml/badge.svg?branch=main)](https://github.com/hashgraph/hedera-block-node/actions/workflows/e2e-tests.yaml) [![codecov](https://codecov.io/github/hashgraph/hedera-block-node/graph/badge.svg?token=OF6T6E8V7U)](https://codecov.io/github/hashgraph/hedera-block-node) @@ -42,7 +54,7 @@ This project is governed by the participating, you are expected to uphold this code of conduct. Please report unacceptable behavior to [oss@hedera.com](mailto:oss@hedera.com). -# 🔐 Security +## 🔐 Security Please do not file a public ticket mentioning the vulnerability. Refer to the security policy defined in the [SECURITY.md](https://github.com/hashgraph/hedera-block-node/blob/main/SECURITY.md). diff --git a/docs/Protocol/README.md b/docs/Protocol/README.md index 8dc715626..1456982b4 100644 --- a/docs/Protocol/README.md +++ b/docs/Protocol/README.md @@ -6,6 +6,6 @@ Each protocol document should describe a single API call and the expected behavior of both sides of that API call, including common error conditions. ## Contents -| Document | API call | Description | -|:---|---:|:---| -| publishBlockStream.md | `publishBlockStream` | The communication between a publisher and a block node when publishing a block stream from an authoritative source such as a consensus node.| +| Document | API call | Description | +|:-----------------------------------------------|---:|:---| +| [publishBlockStream.md](publishBlockStream.md) | `publishBlockStream` | The communication between a publisher and a block node when publishing a block stream from an authoritative source such as a consensus node.| diff --git a/docs/Protocol/publishBlockStream.md b/docs/Protocol/publishBlockStream.md index c1ddefa28..0d607390d 100644 --- a/docs/Protocol/publishBlockStream.md +++ b/docs/Protocol/publishBlockStream.md @@ -1,5 +1,14 @@ # Block Node Connect Protocol for `publishBlockStream` +## Table of Contents +1. [Abstract](#abstract) + 1. [Reliability note](#reliability-note) +1. [Definitions](#definitions) +1. [Base Protocol](#base-protocol) + 1. [Base Protocol Diagram](#base-protocol-diagram) +1. [Error Handling](#error-handling) + 1. [Error Handling Diagram](#error-handling-diagram) + ## Abstract This protocol describes how a Publisher and Block Node SHALL interact for the `publishBlockStream` API. The protocol provides for either side of the diff --git a/docs/README.md b/docs/README.md index 26046cf65..f39c90b19 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,8 +4,9 @@ This document will outline the structure and content of Block Node documentation ## Table of Contents -1. Introduction -2. Concepts +1. [Introduction](#introduction) +1. [Concepts](#concepts) +1. [Release](#release) ## Introduction @@ -15,3 +16,7 @@ This document will outline the structure and content of Block Node documentation ## Concepts - [Hedera Block Node Protocol](Protocol/README.md): Overview of the Hedera Block Node Protocol + +## Release + +- [Release Process Documentation](release.md): Details the steps involved in creating a new release, tagging, and publishing artifacts. diff --git a/docs/configuration.md b/docs/configuration.md deleted file mode 100644 index f35a24fbe..000000000 --- a/docs/configuration.md +++ /dev/null @@ -1,19 +0,0 @@ -# Configuration - -The components of the Hedera Block Node all support loading configuration via the -environment. - -## Default Values - -The default configuration allows users to quickly get up and running without having to configure anything. This provides -ease of use at the trade-off of some insecure default configuration. Most configuration settings have appropriate -defaults and can be left unchanged. It is recommended to browse the properties below and adjust to your needs. - - -| Environment Variable | Description | Default Value | -|-----------------------------------|---------------------------------------------------------------------------------------------------------------|---------------| -| PERSISTENCE_STORAGE_ROOT_PATH | The root path for the storage, if not provided will attempt to create a `data` on the working dir of the app. | | -| CONSUMER_TIMEOUT_THRESHOLD_MILLIS | Time to wait for subscribers before disconnecting in milliseconds | 1500 | -| SERVICE_DELAY_MILLIS | Service shutdown delay in milliseconds | 500 | -| MEDIATOR_RING_BUFFER_SIZE | Size of the ring buffer used by the mediator (must be a power of 2) | 67108864 | -| NOTIFIER_RING_BUFFER_SIZE | Size of the ring buffer used by the notifier (must be a power of 2) | 2048 | diff --git a/docs/release.md b/docs/release.md index 55504a089..cf0c54de4 100644 --- a/docs/release.md +++ b/docs/release.md @@ -1,10 +1,21 @@ -## Release Process Documentation +# Release Process Documentation -### Overview +## Table of Contents + +1. [Overview](#overview) +1. [Kickstart Release Process](#kickstart-release-process) +1. [Release Automation Workflow](#release-automation-workflow) + 1. [Steps](#steps) +1. [Publish Release Workflow](#publish-release-workflow) +1. [Mermaid Diagram](#mermaid-diagram) +1. [Release Meta Process](#release-meta-process) +1. [Conclusion](#conclusion) + +## Overview This document outlines the release process for our project, detailing the steps involved in creating a new release, tagging, and publishing artifacts. -### Kickstart Release Process +## Kickstart Release Process 1. **Milestone and Label Check**: - Ensure all Pull Requests (PRs) have a milestone set and appropriate labels. This is important for generating accurate release notes. @@ -17,14 +28,14 @@ This document outlines the release process for our project, detailing the steps - If the release branch already exists, the workflow should be triggered using the branch release, and the version should be bumped accordingly. - for example: if `release/0.1` exists, and you want to release `0.1.1`, trigger the workflow with `0.1.1` as the semver input and select `release/0.1` on "Use workflow from" dropdown. -### Release Automation Workflow +## Release Automation Workflow The release process is automated using a GitHub Actions workflow (`release-automation.yaml`). The workflow must be triggered manually with a SemVer version input, which specifies the version to be released. The input can be: - `x.n.0-rc1` for release candidates - `x.n.0` for general availability (GA) versions - `x.n.p` for patch versions, including hot-fixes -#### Steps: +### Steps: 1. **Create or Update Release Branch**: - Create a new release branch `release/0.n` if it doesn't exist. @@ -41,7 +52,7 @@ The release process is automated using a GitHub Actions workflow (`release-autom - Upload distribution artifacts to the release. - Publish the artifacts to Sonatype Nexus / Maven Central. -### Publish Release Workflow +## Publish Release Workflow The `Publish Release Workflow` (`publish-release.yaml`) is triggered by the Tag pushed in the previous step. This workflow performs the following tasks: @@ -51,7 +62,7 @@ The `Publish Release Workflow` (`publish-release.yaml`) is triggered by the Tag 2. **Helm Chart Publishing**: - Publish a new Helm chart with the same version to the custom GitHub Pages of the repo in the `Charts` folder. -### Mermaid Diagram +## Mermaid Diagram ```mermaid graph TD @@ -84,7 +95,7 @@ graph TD end ``` -### Release Meta Process +## Release Meta Process The meta process typically follows these steps: @@ -116,4 +127,3 @@ graph TD ## Conclusion This release process ensures that all steps are automated, from checking PR milestones and labels to creating branches, tagging releases, and publishing Docker images and Helm charts. By following this documentation, you can streamline the release workflow and maintain consistency across all releases. - diff --git a/server/README.md b/server/README.md index 9517d1d20..826b485cc 100644 --- a/server/README.md +++ b/server/README.md @@ -1,5 +1,15 @@ # Block Node Application +## Table of Contents + +1. [Overview](#overview) +1. [Prerequisites](#prerequisites) +1. [Configuration](#configuration) +1. [Metrics](#metrics) +1. [Design](#design) + 1. [Block Persistence](#block-persistence) + 1. [Bi-directional Producer/Consumer Streaming with gRPC](#bi-directional-producerconsumer-streaming-with-grpc) + ## Overview The Block Stream Application is designed handle the output of a Hedera Node, which would be in form of Block Stream. @@ -15,3 +25,17 @@ It uses various configuration sources and dependency injection to manage its com ## Configuration Refer to the [Configuration](docs/configuration.md) for configuration options. + +## Metrics + +Refer to the [Metrics](docs/metrics.md) for metrics available in the system. + +## Design + +### Block Persistence + +Refer to the [Block Persistence](docs/design/block-persistence.md) for details on how blocks are persisted. + +### Bi-directional Producer/Consumer Streaming with gRPC + +Refer to the [Bi-directional Producer/Consumer Streaming with gRPC](docs/design/bidi-producer-consumers-streaming.md) for details on how the gRPC streaming is implemented. diff --git a/server/docs/configuration.md b/server/docs/configuration.md index f35a24fbe..39d4b97af 100644 --- a/server/docs/configuration.md +++ b/server/docs/configuration.md @@ -1,4 +1,4 @@ -# Configuration +# Server Configuration The components of the Hedera Block Node all support loading configuration via the environment. diff --git a/server/docs/design/bidi-producer-consumers-streaming.md b/server/docs/design/bidi-producer-consumers-streaming.md index 08215d405..136c680e8 100644 --- a/server/docs/design/bidi-producer-consumers-streaming.md +++ b/server/docs/design/bidi-producer-consumers-streaming.md @@ -1,5 +1,29 @@ # Bi-directional Producer/Consumer Streaming with gRPC +## Table of Contents + +1. [Purpose](#purpose) +1. [Goals](#goals) +1. [Terms](#terms) +1. [Block Node gRPC Streaming Services API](#block-node-grpc-streaming-services-api) +1. [Approaches](#approaches) + 1. [Approach 1: Directly passing BlockItems from `ProducerBlockItemObserver` to N `ConsumerBlockItemObserver`s](#approach-1-directly-passing-blockitems-from-producerblockitemobserver-to-n-consumerblockitemobservers) + 1. [Approach 2: Use a shared data structure between `ProducerBlockItemObserver` and `ConsumerBlockItemObserver`s. Consumers busy-wait for new BlockItems](#approach-2-use-a-shared-data-structure-between-producerblockitemobserver-and-consumerblockitemobservers-consumers-busy-wait-for-new-blockitems) + 1. [Approach 3: Use a shared data structure between `ProducerBlockItemObserver` and `ConsumerBlockItemObserver`s. Use downstream consumer BlockItemResponses to drive the process of sending new BlockItems](#approach-3-use-a-shared-data-structure-between-producerblockitemobserver-and-consumerblockitemobservers-use-downstream-consumer-blockitemresponses-to-drive-the-process-of-sending-new-blockitems) + 1. [Approach 4: Shared data structure between producer and consumer services. Leveraging the LMAX Disruptor library to manage inter-process pub/sub message-passing between producer and consumers via RingBuffer](#approach-4-shared-data-structure-between-producer-and-consumer-services-leveraging-the-lmax-disruptor-library-to-manage-inter-process-pubsub-message-passing-between-producer-and-consumers-via-ringbuffer) +1. [Design](#design) + 1. [Producer Registration Flow](#producer-registration-flow) + 1. [Consumer Registration Flow](#consumer-registration-flow) + 1. [Runtime Streaming](#runtime-streaming) + 1. [Entities](#entities) +1. [Diagrams](#diagrams) + 1. [Producer Registration Flow](#producer-registration-flow-1) + 1. [Consumer Registration Flow](#consumer-registration-flow-1) + 1. [Class Diagram of all Entities and their Relationships](#class-diagram-of-all-entities-and-their-relationships) + 1. [Runtime Stream of BlockItems from Producer to Consumers](#runtime-stream-of-blockitems-from-producer-to-consumers) + +--- + ## Purpose A primary use case of the `hedera-block-node` is to stream live BlockItems (see Terms section) from a producer @@ -25,7 +49,7 @@ point for custom logic is an implementation of `GrpcService`. --- -### Terms +## Terms **BlockItem** - The BlockItem is the primary data structure passed between the producer, the `hedera-block-node` and consumers. A defined sequence of BlockItems represent a Block when stored on the `hedera-block-node`. @@ -51,14 +75,13 @@ unsubscribed from a producer so that internal objects can be cleaned up and reso --- -### Block Node gRPC Streaming Services API +## Block Node gRPC Streaming Services API The Block Node gRPC Streaming Services API is now aligned with the names and simplified types defined in the [`hedera-protobufs` repository on the `continue-block-node` branch](https://github.com/hashgraph/hedera-protobufs/blob/25783427575ded59d06d6bf1ed253fd24ef3c437/block/block_service.proto#L701-L742). --- - ## Approaches: All the following approaches require integrating with Helidon 4.x.x gRPC services to implement the bidirectional @@ -80,7 +103,7 @@ BlockItem to the downstream consumer. Helidon invokes `ConsumerBlockItemObserver from the consumer in receipt of BlockItems. -### Approach 1: Directly passing BlockItems from `ProducerBlockItemObserver` to N `ConsumerBlockItemObserver`s. +### Approach 1: Directly passing BlockItems from `ProducerBlockItemObserver` to N `ConsumerBlockItemObserver`s. Directly passing BlockItems from the `ProducerBlockItemObserver` to N `ConsumerBlockItemObserver`s without storing BlockItems in an intermediate data structure. This approach was the basis for one of the first implementations of gRPC @@ -117,7 +140,6 @@ Drawbacks: downstream consumer in a timely way. The aggressive behavior of the busy-wait could complicate future use cases requiring downstream consumer response processing. - ### Approach 3: Use a shared data structure between `ProducerBlockItemObserver` and `ConsumerBlockItemObserver`s. Use downstream consumer BlockItemResponses to drive the process of sending new BlockItems. With this approach, the `ProducerBlockItemObserver` will store BlockItems in a shared data structure before immediately @@ -193,7 +215,6 @@ the `ResponseStreamObserver` managed by Helidon for transmitting BlockItemRespon `BlockItemStreamService` will also subscribe the `ConsumerBlockItemObserver` to the `StreamMediator` to receive the streaming BlockItems from the producer. - ### Runtime Streaming At runtime, the `ProducerBlockItemObserver` will receive the latest BlockItem from the producer via Helidon and will @@ -250,20 +271,18 @@ the latest BlockItem to the ConsumerBlockItemObserver when it is available in th **BlockPersistenceHandler** - The BlockPersistenceHandler is responsible for writing the latest BlockItem to disk. --- -## Diagrams +## Diagrams ### Producer Registration Flow ![Producer Registration](assets/00036-producer-registration.png) - ### Consumer Registration Flow ![Consumer Registration](assets/00036-consumer-registration.png) - -### Class Diagram of all Entities and their Relationships +### Class Diagram of all Entities and their Relationships ![Class Diagram](assets/00036-demo-disruptor-class-diagram.png) @@ -271,6 +290,4 @@ the latest BlockItem to the ConsumerBlockItemObserver when it is available in th ![Sequence Diagram](assets/00036-refactor-demo-disruptor.png) - --- - diff --git a/server/docs/design/block-persistence.md b/server/docs/design/block-persistence.md index ea10f49ac..561c9b428 100644 --- a/server/docs/design/block-persistence.md +++ b/server/docs/design/block-persistence.md @@ -1,5 +1,15 @@ # Block Persistence +## Table of Contents + +1. [Purpose](#purpose) +1. [Goals](#goals) +1. [Terms](#terms) +1. [Entities](#entities) +1. [Design](#design) + +--- + ## Purpose The main objective of the `hedera-block-node` project is to replace the storage of Consensus Node artifacts (e.g. @@ -9,7 +19,7 @@ when they arise. --- -### Goals +## Goals 1) BlockItems streamed from a producer (e.g. Consensus Node) must be collated and persisted as a Block. Per the specification, a Block is an ordered list of BlockItems. How the Block is persisted is an implementation detail. @@ -17,7 +27,7 @@ when they arise. --- -### Terms +## Terms **BlockItem** - A BlockItem is the primary data structure passed between the producer, the `hedera-block-node` and consumers. The BlockItem description and protobuf definition are maintained in the `hedera-protobuf` @@ -29,7 +39,7 @@ BlockItems. The Block description and protobuf definition are maintained in the --- -### Entities +## Entities **BlockReader** - An interface defining methods used to read a Block from storage. It represents a lower-level component whose implementation is directly responsible for reading a Block from storage. @@ -42,7 +52,7 @@ component whose implementation is directly responsible for removing a Block from --- -### Design +## Design The design for `Block` persistence is fairly straightforward. Block server objects should use the persistence entity interfaces to read, write and remove `Block`s from storage. `BlockItem`s streamed from a producer are read off the wire diff --git a/server/docs/metrics.md b/server/docs/metrics.md index 37c7277d8..ca71aa28d 100644 --- a/server/docs/metrics.md +++ b/server/docs/metrics.md @@ -1,9 +1,19 @@ # Metrics This document describes the metrics that are available in the system, its purpose, and how to use them. +## Table of Contents + +1. [Overview](#overview) +1. [Purpose](#purpose) +1. [Configuration](#configuration) + 1. [Prometheus](#prometheus) +1. [Usage](#usage) + 1. [Local Development](#local-development) +1. [Existing Metrics](#existing-metrics) + ## Overview -We are using Prometheus to collect metrics from the system, through the use of [Hedera Platform SDK - Metrics](https://github.com/hashgraph/hedera-services/blob/develop/platform-sdk/docs/base/metrics/metrics.md). +We are using Prometheus to collect metrics from the system, through the use of [Hedera Platform SDK - Metrics](https://github.com/hashgraph/hedera-services/blob/develop/platform-sdk/docs/base/metrics/metrics.md). ## Purpose @@ -22,6 +32,7 @@ Prefix: prometheus, ie. `prometheus.configKey` | endpointMaxBacklogAllowed | The maximum number of incoming TCP connections which the system will queue internally | 1 | ## Usage + All classes that need to observe metrics can get them from the BlockNodeContext. If a new metric is needed, it can be added to MetricsService, and then used in the classes that need it. This is to avoid having to define the metrics in several places. @@ -30,6 +41,7 @@ MetricsService centralizes the creation of metrics and provides a way to access To check the metrics you can access the Prometheus endpoint at `http://localhost:9999/metrics`. ### Local Development + For developers, when using the gradle task `startDockerContainer` it will automatically start a prometheus and grafana services preconfigured locally with credentials: username `admin` and password `admin` and the dashboard already provisioned with the current metrics and widgets. Dashboard is called `Block-Node Server Dashboard` and its source is kept on folder: `server/docker/metrics/dashboards` as: `block-node-server.json`. @@ -38,7 +50,6 @@ When doing changes to the dashboard on grafana is important to copy the json to If needed to create another dashboard is possible to include it by adding it to the same folder. - ## Existing Metrics All metrics have `hedera_block_node` prefix. @@ -49,4 +60,3 @@ All metrics have `hedera_block_node` prefix. | blocks_persisted | the number of blocks persisted | Counter | | subscribers | The number of subscribers | Gauge | | single_blocks_retrieved | the number of single blocks requested | Counter | - From c24f712c689d5eaf37e99f1cd2e5f985d6ebc299 Mon Sep 17 00:00:00 2001 From: Atanas Atanasov Date: Fri, 1 Nov 2024 12:41:09 +0200 Subject: [PATCH 04/11] adding quickstart to server Signed-off-by: Atanas Atanasov --- README.md | 29 ++++++++++++++-------- server/README.md | 10 +++----- server/docs/quickstart.md | 52 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 75 insertions(+), 16 deletions(-) create mode 100644 server/docs/quickstart.md diff --git a/README.md b/README.md index 6a4311c43..d4bd0f3f8 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,9 @@ Implementation of the Hedera Block Node, which is responsible for consuming the ## Table of Contents -1. [Overview of child modules](#overview-of-child-modules) 1. [Project Links](#project-links) +1. [Prerequisites](#prerequisites) +1. [Overview of child modules](#overview-of-child-modules) 1. [Getting Started](#getting-started) 1. [Support](#support) 1. [Contributing](#contributing) @@ -13,14 +14,6 @@ Implementation of the Hedera Block Node, which is responsible for consuming the 1. [Security](#-security) 1. [License](#license) -## Overview of child modules - -- `server`: implementation of the block node, which contains the main application and all the necessary code to run the block node. -- `simulator`: A simulator for the block node, which can be used to test the block node in a local environment. -- `common`: Module responsible for holding common literals, utilities and types used by the other modules. -- `suites`: A set of e2e tests that can be used to verify the correctness of the block node. -- `tools`: A set of command line tools for working with block stream files. - ## Project Links [![Build Application](https://github.com/hashgraph/hedera-block-node/actions/workflows/build-application.yaml/badge.svg?branch=main)](https://github.com/hashgraph/hedera-block-node/actions/workflows/build-application.yaml) @@ -29,9 +22,25 @@ Implementation of the Hedera Block Node, which is responsible for consuming the [![Latest Version](https://img.shields.io/github/v/tag/hashgraph/hedera-block-node?sort=semver&label=version)](README.md) [![Made With](https://img.shields.io/badge/made_with-java-blue)](https://github.com/hashgraph/hedera-block-node/) -[![Development Branch](https://img.shields.io/badge/docs-quickstart-green.svg)](docs/gradle-quickstart.md) +[![Development Branch](https://img.shields.io/badge/docs-quickstart-green.svg)](docs/README.md) [![License](https://img.shields.io/badge/license-apache2-blue.svg)](LICENSE) +## Prerequisites + +- Java 21 (temurin recommended) +- Gradle (using the wrapper `./gradlew` is highly recommended) +- Docker (recommended for running the projects) +- IntelliJ IDEA (recommended for development) + +## Overview of child modules + +- `server`: implementation of the block node, which contains the main application and all the necessary code to run the block node. +- `simulator`: A simulator for the block node, which can be used to test the block node in a local environment. +- `common`: Module responsible for holding common literals, utilities and types used by the other modules. +- `suites`: A set of e2e tests that can be used to verify the correctness of the block node. +- `tools`: A set of command line tools for working with block stream files. + + ## Getting Started Refer to the [Hedera Block Node Documentation Overview](docs/README.md) for more information about the project, design and guides. diff --git a/server/README.md b/server/README.md index 826b485cc..281ac332f 100644 --- a/server/README.md +++ b/server/README.md @@ -16,16 +16,14 @@ The Block Stream Application is designed handle the output of a Hedera Node, whi By handling we can understand verifying, storing and applying needed state changes. It uses various configuration sources and dependency injection to manage its components. -## Prerequisites - -- Java 21 -- Gradle -- IntelliJ IDEA (recommended for development) - ## Configuration Refer to the [Configuration](docs/configuration.md) for configuration options. +## Quickstart + +Refer to the [Quickstart](docs/quickstart.md) for a quick guide on how to get started with the application. + ## Metrics Refer to the [Metrics](docs/metrics.md) for metrics available in the system. diff --git a/server/docs/quickstart.md b/server/docs/quickstart.md new file mode 100644 index 000000000..0bab51caa --- /dev/null +++ b/server/docs/quickstart.md @@ -0,0 +1,52 @@ +# Quickstart of the Server + +## Table of Contents +1. [Configuration](#configuration) +1. [Running locally](#running-locally) + 1. [Build the Server](#build-the-server) + 1. [Run the Server](#run-the-server) + 1. [Run the Server with Debug](#run-the-server-with-debug) + 1. [Stop the Server](#stop-the-server) + +## Configuration + +Refer to the [Configuration](configuration.md) for configuration options. + +## Running locally: + +Assuming your working directory is the repo root: + +### Build the Server + +1. To quickly build the Server sources (without running tests), do the following: + ```bash + ./gradlew clean build -x test + ``` + +1. To build the Server docker image, do the following: + ```bash + ./gradlew createDockerImage + ``` + +### Run the Server + +1. To start the Server, do the following: + ```bash + ./gradlew startDockerContainer + ``` + +### Run the Server with Debug + +1. To start the Server with debug enabled, do the following: + ```bash + ./gradlew startDockerDebugContainer + ``` + +1. Attach your remote jvm debugger to port 5005. + +### Stop the Server + +1. To stop the Server do the following: + ```bash + ./gradlew stopDockerContainer + ``` From 82e636cff06403c60e75fb564cdfa299a7ff8bc6 Mon Sep 17 00:00:00 2001 From: Atanas Atanasov Date: Fri, 1 Nov 2024 13:39:55 +0200 Subject: [PATCH 05/11] adding quickstart to simulator Signed-off-by: Atanas Atanasov --- server/docs/quickstart.md | 24 ++++++++++---- simulator/README.md | 38 +++++---------------- simulator/docs/configuration.md | 19 +++++------ simulator/docs/quickstart.md | 58 +++++++++++++++++++++++++++++++++ 4 files changed, 93 insertions(+), 46 deletions(-) create mode 100644 simulator/docs/quickstart.md diff --git a/server/docs/quickstart.md b/server/docs/quickstart.md index 0bab51caa..305aca4d7 100644 --- a/server/docs/quickstart.md +++ b/server/docs/quickstart.md @@ -1,6 +1,7 @@ # Quickstart of the Server ## Table of Contents + 1. [Configuration](#configuration) 1. [Running locally](#running-locally) 1. [Build the Server](#build-the-server) @@ -14,32 +15,43 @@ Refer to the [Configuration](configuration.md) for configuration options. ## Running locally: -Assuming your working directory is the repo root: +- Server subproject qualifier: `:server` +- Assuming your working directory is the repo root + +> **NOTE:** using the `-p` flag for `./gradlew` so not to specify the target subproject +> on each task when running multiple tasks, but using the qualifier `:server:` when +> we have only one task to run. ### Build the Server +> **NOTE:** if you have not done so already, it is +> generally recommended to build the entire repo first: +> ```bash +> ./gradlew clean build -x test +> ``` + 1. To quickly build the Server sources (without running tests), do the following: ```bash - ./gradlew clean build -x test + ./gradlew -p server clean build -x test ``` 1. To build the Server docker image, do the following: ```bash - ./gradlew createDockerImage + ./gradlew :server:createDockerImage ``` ### Run the Server 1. To start the Server, do the following: ```bash - ./gradlew startDockerContainer + ./gradlew :server:startDockerContainer ``` ### Run the Server with Debug 1. To start the Server with debug enabled, do the following: ```bash - ./gradlew startDockerDebugContainer + ./gradlew :server:startDockerDebugContainer ``` 1. Attach your remote jvm debugger to port 5005. @@ -48,5 +60,5 @@ Assuming your working directory is the repo root: 1. To stop the Server do the following: ```bash - ./gradlew stopDockerContainer + ./gradlew :server:stopDockerContainer ``` diff --git a/simulator/README.md b/simulator/README.md index f41f63c94..e2e64052b 100644 --- a/simulator/README.md +++ b/simulator/README.md @@ -1,16 +1,16 @@ # Block Stream Simulator +## Table of Contents +1. [Overview](#overview) +1. [Project Design Structure](#project-design-structure) +1. [Configuration](#configuration) +1. [Quickstart](#quickstart) + ## Overview The Block Stream Simulator is designed to simulate block streaming for Hedera Hashgraph. It uses various configuration sources and dependency injection to manage its components. -## Prerequisites - -- Java 21 -- Gradle -- IntelliJ IDEA (recommended for development) - ## Project Design Structure Uses Dagger2 for dependency injection, the project has a modular structure and divides the Dagger dependencies into modules, but all modules used can be found at the root Injection Module: @@ -33,28 +33,6 @@ The BlockStreamSimulatorApp consumes other services that are injected using Dagg Refer to the [Configuration](docs/configuration.md) for configuration options. -## Building the Project - -To build the project, run the following command: - -```sh -./gradlew :simulator:build -``` - -## Running the Project +## Quickstart -Usually you will want to run a Block-Node server before the simulator, for that you can use the following commnad: - -```sh - ./gradlew :server:run -``` -However we recommend running the block-node server as a docker container: -```sh -./gradlew :server:build :server:createDockerImage :server:startDockerContainer -``` - -Once the project is built, you can run it using the following command: - -```sh -./gradlew :simulator:run -``` +Refer to the [Quickstart](docs/quickstart.md) for a quick guide on how to get started with the application. diff --git a/simulator/docs/configuration.md b/simulator/docs/configuration.md index 36bc2ea97..f05bf5b2f 100644 --- a/simulator/docs/configuration.md +++ b/simulator/docs/configuration.md @@ -1,10 +1,11 @@ -## Configuration +# Configuration -There are 2 configuration sets: -1. BlockStreamConfig: contains the configuration for the Block Stream Simulator logic and the generation module. -2. GrpcConfig: contains the configuration for the gRPC communication with the Block-Node. +There are 3 configuration sets: +1. [BlockStreamConfig](#blockstreamconfig): contains the configuration for the Block Stream Simulator logic. +1. [GeneratorConfig](#generatorconfig): contains the configuration for the Block Stream Simulator generation module. +1. [GrpcConfig](#grpcconfig): contains the configuration for the gRPC communication with the Block-Node. -### BlockStreamConfig +## BlockStreamConfig Uses the prefix `blockStream` so all properties should start with `blockStream.` | Key | Description | Default Value | @@ -15,7 +16,7 @@ Uses the prefix `blockStream` so all properties should start with `blockStream.` | `millisecondsPerBlock` | if streamingMode is `MILLIS_PER_BLOCK` this will be the time to wait between blocks in milliseconds | `1_000` | | `blockItemsBatchSize` | the number of block items to send in a single batch, however if a block has less block items, it will send all the items in a block | `1_000` | -### GeneratorConfig +## GeneratorConfig Uses the prefix `generator` so all properties should start with `generator.` | Key | Description | Default Value | |--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------| @@ -25,12 +26,10 @@ Uses the prefix `generator` so all properties should start with `generator.` | `paddedLength` | on the `BlockAsFileLargeDataSets` implementation, the length of the padded left zeroes `000001.blk.gz` | 36 | | `fileExtension` | on the `BlockAsFileLargeDataSets` implementation, the extension of the files to be streamed | `.blk.gz` | - - -### GrpcConfig +## GrpcConfig Uses the prefix `grpc` so all properties should start with `grpc.` | Key | Description | Default Value | |-----------------|----------------------------|---------------| | `serverAddress` | The host of the Block-Node | `localhost` | -| `port` | The port of the Block-Node | `8080` | \ No newline at end of file +| `port` | The port of the Block-Node | `8080` | diff --git a/simulator/docs/quickstart.md b/simulator/docs/quickstart.md new file mode 100644 index 000000000..36905a261 --- /dev/null +++ b/simulator/docs/quickstart.md @@ -0,0 +1,58 @@ +# Quickstart of the Simulator + +## Table of Contents + +1. [Configuration](#configuration) +1. [Running locally](#running-locally) + 1. [Build the Simulator](#build-the-simulator) + 1. [Run the Server first](#run-the-server-first) + 1. [Run the Simulator](#run-the-simulator) + 1. [Run the Simulator with Debug](#run-the-simulator-with-debug) + +## Configuration + +Refer to the [Configuration](configuration.md) for configuration options. + +## Running locally: + +- Simulator subproject qualifier:`:simulator` +- Assuming your working directory is the repo root + +> **NOTE:** using the `-p` flag for `./gradlew` so not to specify the target subproject +> on each task when running multiple tasks, but using the qualifier `:simulator:` when +> we have only one task to run. + +### Build the Simulator + +> **NOTE:** if you have not done so already, it is +> generally recommended to build the entire repo first: +> ```bash +> ./gradlew clean build -x test +> ``` + +1. To quickly build the Simulator sources (without running tests), do the following: + ```bash + ./gradlew -p simulator clean build -x test + ``` + +### Run the Server first + +Usually, you would want to run the [Server](../../server/README.md) first, refer to the +[Quickstart of the Server](../../server/docs/quickstart.md) for a quick guide on how to +get started with the application. + +### Run the Simulator + +1. To start the Simulator, do the following: + ```bash + ./gradlew :simulator:run + ``` + +### Run the Simulator with Debug + +1. To start the Simulator with debug enabled, do the following: + ```bash + ./gradlew :simulator:run --debug-jvm + ``` + +1. Attach your remote jvm debugger to port 5005. From 0dc00fe2119e352ce30e0b4dac800ce34281540c Mon Sep 17 00:00:00 2001 From: Atanas Atanasov Date: Fri, 1 Nov 2024 14:14:07 +0200 Subject: [PATCH 06/11] redundant license change Signed-off-by: Atanas Atanasov --- LICENSE | 1 + 1 file changed, 1 insertion(+) diff --git a/LICENSE b/LICENSE index 261eeb9e9..57bc88a15 100644 --- a/LICENSE +++ b/LICENSE @@ -199,3 +199,4 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. + From 2b60c3aefc76f70fd49f2dfe41ef34965f6fd5be Mon Sep 17 00:00:00 2001 From: Atanas Atanasov Date: Mon, 4 Nov 2024 17:28:05 +0200 Subject: [PATCH 07/11] tools now fit the new styling Signed-off-by: Atanas Atanasov --- tools/README.md | 31 +++++++++++++++++++++---------- tools/docs/quickstart.md | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 10 deletions(-) create mode 100644 tools/docs/quickstart.md diff --git a/tools/README.md b/tools/README.md index 5def135d9..10658f7e8 100644 --- a/tools/README.md +++ b/tools/README.md @@ -1,21 +1,27 @@ # Command Line Tools for Block Nodes & Streams +## Table of Contents + +1. [Overview](#overview) +1. [Subcommands](#subcommands) + 1. [The `json` Subcommand](#the-json-subcommand) + 1. [The `info` Subcommand](#the-info-subcommand) +1. [Running from command line](#running-from-command-line) + +## Overview + This subproject provides command line tools for working with block stream files and maybe other things in the future. It uses [picocli](https://picocli.info) to provide a command line interface which makes it easy to extend and add new subcommands or options. -## Running from command line -You can run through gradle with the `tools:run` task. For example, to see the help for the `info` subcommand, you can -run: - -`./gradlew -q tools:run --args="info --help"` - ## Subcommands + The following subcommands are available: -- **json** Converts a binary block stream to JSON -- **info** Prints info for block files +- `json` - Converts a binary block stream to JSON +- `info` - Prints info for block files + +### The `json` Subcommand -# JSON Subcommand Converts a binary block stream to JSON `Usage: subcommands json [-t] [-ms=] [...]` @@ -29,7 +35,8 @@ transactions human-readable. - `...` - The block files or directories of block files to convert to JSON -# Info Subcommand +### The `info` Subcommand + Prints info for block files `Usage: subcommands info [-c] [-ms=] [-o=] [...]` @@ -43,3 +50,7 @@ Prints info for block files - Output to file rather than stdout - `...` - The block files or directories of block files to print info for + +## Running from command line + +Refer to the [Quickstart](docs/quickstart.md) for a quick guide on how to run the tools CLI. diff --git a/tools/docs/quickstart.md b/tools/docs/quickstart.md new file mode 100644 index 000000000..ddb28735b --- /dev/null +++ b/tools/docs/quickstart.md @@ -0,0 +1,39 @@ +# Quickstart of the Tools + +## Table of Contents + +1. [Running locally](#running-locally) + 1. [Build the Tools](#build-the-tools) + 1. [Run the Tools](#run-the-tools) + +## Running locally: + +- Tools subproject qualifier: `:tools` +- Assuming your working directory is the repo root + +> **NOTE:** using the `-p` flag for `./gradlew` so not to specify the target subproject +> on each task when running multiple tasks, but using the qualifier `:tools:` when +> we have only one task to run. + +### Build the Tools + +> **NOTE:** if you have not done so already, it is +> generally recommended to build the entire repo first: +> ```bash +> ./gradlew clean build -x test +> ``` + +1. To quickly build the Tools sources (without running tests), do the following: + ```bash + ./gradlew -p tools clean build -x test + ``` + +### Run the Tools + +1. To run the Tools, do the following: + ```bash + # Here is an example of running the info command with the help option, simply + # replace `info --help` with the desired command and options to run the tools + # quickly using the `./gradlew run` task. + ./gradlew -q :tools:run --args="info --help" + ``` From c33d48dab3a906285341fe48964ff6e0c5ed8d94 Mon Sep 17 00:00:00 2001 From: Atanas Atanasov Date: Tue, 5 Nov 2024 09:55:20 +0200 Subject: [PATCH 08/11] addressing a pr comment for readability Signed-off-by: Atanas Atanasov --- server/docs/quickstart.md | 8 +++++--- simulator/docs/quickstart.md | 8 +++++--- tools/docs/quickstart.md | 8 +++++--- 3 files changed, 15 insertions(+), 9 deletions(-) diff --git a/server/docs/quickstart.md b/server/docs/quickstart.md index 305aca4d7..c042a1261 100644 --- a/server/docs/quickstart.md +++ b/server/docs/quickstart.md @@ -18,9 +18,11 @@ Refer to the [Configuration](configuration.md) for configuration options. - Server subproject qualifier: `:server` - Assuming your working directory is the repo root -> **NOTE:** using the `-p` flag for `./gradlew` so not to specify the target subproject -> on each task when running multiple tasks, but using the qualifier `:server:` when -> we have only one task to run. +> **NOTE:** one may use the `-p` flag for `./gradlew` in order to avoid +> specifying the target subproject repeatedly on each task when running +> multiple tasks. When running only a single task, however, it is +> recommended to use the project qualifier (i.e. `:server:`) for +> both simplicity and clarity. ### Build the Server diff --git a/simulator/docs/quickstart.md b/simulator/docs/quickstart.md index 36905a261..0a5aff3a5 100644 --- a/simulator/docs/quickstart.md +++ b/simulator/docs/quickstart.md @@ -18,9 +18,11 @@ Refer to the [Configuration](configuration.md) for configuration options. - Simulator subproject qualifier:`:simulator` - Assuming your working directory is the repo root -> **NOTE:** using the `-p` flag for `./gradlew` so not to specify the target subproject -> on each task when running multiple tasks, but using the qualifier `:simulator:` when -> we have only one task to run. +> **NOTE:** one may use the `-p` flag for `./gradlew` in order to avoid +> specifying the target subproject repeatedly on each task when running +> multiple tasks. When running only a single task, however, it is +> recommended to use the project qualifier (i.e. `:simulator:`) for +> both simplicity and clarity. ### Build the Simulator diff --git a/tools/docs/quickstart.md b/tools/docs/quickstart.md index ddb28735b..1a5ffdea0 100644 --- a/tools/docs/quickstart.md +++ b/tools/docs/quickstart.md @@ -11,9 +11,11 @@ - Tools subproject qualifier: `:tools` - Assuming your working directory is the repo root -> **NOTE:** using the `-p` flag for `./gradlew` so not to specify the target subproject -> on each task when running multiple tasks, but using the qualifier `:tools:` when -> we have only one task to run. +> **NOTE:** one may use the `-p` flag for `./gradlew` in order to avoid +> specifying the target subproject repeatedly on each task when running +> multiple tasks. When running only a single task, however, it is +> recommended to use the project qualifier (i.e. `:tools:`) for +> both simplicity and clarity. ### Build the Tools From c5ddda1540cadeda4378bda4a1d27674d52ddda5 Mon Sep 17 00:00:00 2001 From: Atanas Atanasov Date: Thu, 7 Nov 2024 11:12:25 +0200 Subject: [PATCH 09/11] addressing comments for maintainability Signed-off-by: Atanas Atanasov --- docs/Protocol/README.md | 6 +-- docs/release.md | 49 +++++++++---------- server/README.md | 4 +- server/docs/configuration.md | 15 +++--- .../bidi-producer-consumers-streaming.md | 34 ++++++------- server/docs/metrics.md | 22 ++++----- server/docs/quickstart.md | 30 ++++++------ simulator/README.md | 11 +++-- simulator/docs/configuration.md | 40 ++++++++------- simulator/docs/quickstart.md | 18 +++---- tools/README.md | 21 +++++--- tools/docs/quickstart.md | 18 +++---- 12 files changed, 139 insertions(+), 129 deletions(-) diff --git a/docs/Protocol/README.md b/docs/Protocol/README.md index 1456982b4..3b7191fdb 100644 --- a/docs/Protocol/README.md +++ b/docs/Protocol/README.md @@ -6,6 +6,6 @@ Each protocol document should describe a single API call and the expected behavior of both sides of that API call, including common error conditions. ## Contents -| Document | API call | Description | -|:-----------------------------------------------|---:|:---| -| [publishBlockStream.md](publishBlockStream.md) | `publishBlockStream` | The communication between a publisher and a block node when publishing a block stream from an authoritative source such as a consensus node.| +| Document | API call | Description | +|:---|---:|:---| +| [publishBlockStream.md](publishBlockStream.md) | `publishBlockStream` | The communication between a publisher and a block node when publishing a block stream from an authoritative source such as a consensus node. | diff --git a/docs/release.md b/docs/release.md index cf0c54de4..ce86b0900 100644 --- a/docs/release.md +++ b/docs/release.md @@ -5,7 +5,7 @@ 1. [Overview](#overview) 1. [Kickstart Release Process](#kickstart-release-process) 1. [Release Automation Workflow](#release-automation-workflow) - 1. [Steps](#steps) + 1. [Steps](#steps) 1. [Publish Release Workflow](#publish-release-workflow) 1. [Mermaid Diagram](#mermaid-diagram) 1. [Release Meta Process](#release-meta-process) @@ -18,14 +18,14 @@ This document outlines the release process for our project, detailing the steps ## Kickstart Release Process 1. **Milestone and Label Check**: - - Ensure all Pull Requests (PRs) have a milestone set and appropriate labels. This is important for generating accurate release notes. - - There is a CI check to ensure that all PRs have a milestone and labels set. If not, the PR will not be merged. - - Make sure all PRs and Issues related to the related milestone are already closed, if not closed them before proceeding or move them to the next milestone. + - Ensure all Pull Requests (PRs) have a milestone set and appropriate labels. This is important for generating accurate release notes. + - There is a CI check to ensure that all PRs have a milestone and labels set. If not, the PR will not be merged. + - Make sure all PRs and Issues related to the related milestone are already closed, if not closed them before proceeding or move them to the next milestone. 2. **Trigger Release Automation Workflow**: - - Manually trigger the `release-automation.yaml` workflow with the desired version to be released. - - The workflow will create a new release branch, bump the version, and tag the release. - - The release branch will be used for any patch versions or hot-fixes. - - If the release branch already exists, the workflow should be triggered using the branch release, and the version should be bumped accordingly. + - Manually trigger the `release-automation.yaml` workflow with the desired version to be released. + - The workflow will create a new release branch, bump the version, and tag the release. + - The release branch will be used for any patch versions or hot-fixes. + - If the release branch already exists, the workflow should be triggered using the branch release, and the version should be bumped accordingly. - for example: if `release/0.1` exists, and you want to release `0.1.1`, trigger the workflow with `0.1.1` as the semver input and select `release/0.1` on "Use workflow from" dropdown. ## Release Automation Workflow @@ -38,29 +38,29 @@ The release process is automated using a GitHub Actions workflow (`release-autom ### Steps: 1. **Create or Update Release Branch**: - - Create a new release branch `release/0.n` if it doesn't exist. - - If the release branch was created, means it's a new release, create a PR on the main branch to increase the snapshot version to the next version. - - Bump the version to the input provided in the release branch. + - Create a new release branch `release/0.n` if it doesn't exist. + - If the release branch was created, means it's a new release, create a PR on the main branch to increase the snapshot version to the next version. + - Bump the version to the input provided in the release branch. 2. **Tagging**: - - After bumping the version on the release branch, tag the version as `vX.n.p` (where `X.n.p` is the input). - - Push the tag, triggering the `Publish Release Workflow`. + - After bumping the version on the release branch, tag the version as `vX.n.p` (where `X.n.p` is the input). + - Push the tag, triggering the `Publish Release Workflow`. 3. **Release Notes and Milestone**: - - Create release notes for the new Tag. - - Close the milestone associated with the release. - - Upload distribution artifacts to the release. - - Publish the artifacts to Sonatype Nexus / Maven Central. + - Create release notes for the new Tag. + - Close the milestone associated with the release. + - Upload distribution artifacts to the release. + - Publish the artifacts to Sonatype Nexus / Maven Central. ## Publish Release Workflow The `Publish Release Workflow` (`publish-release.yaml`) is triggered by the Tag pushed in the previous step. This workflow performs the following tasks: 1. **Docker Image Creation and Publishing**: - - Create a Docker image with the new release version and publish it to the GitHub Container Registry (`ghcr.io`). + - Create a Docker image with the new release version and publish it to the GitHub Container Registry (`ghcr.io`). 2. **Helm Chart Publishing**: - - Publish a new Helm chart with the same version to the custom GitHub Pages of the repo in the `Charts` folder. + - Publish a new Helm chart with the same version to the custom GitHub Pages of the repo in the `Charts` folder. ## Mermaid Diagram @@ -100,15 +100,15 @@ graph TD The meta process typically follows these steps: 1. **Initial Release Candidate (RC)**: - - Trigger the Release Automation Workflow with an `rc` version (e.g., `x.n.0-rcY`). - - Perform integration and performance testing on this version. + - Trigger the Release Automation Workflow with an `rc` version (e.g., `x.n.0-rcY`). + - Perform integration and performance testing on this version. 2. **General Availability (GA)**: - - Once testing is successful, trigger the Release Automation Workflow again for the same version but as a GA version (e.g., `x.n.0`). + - Once testing is successful, trigger the Release Automation Workflow again for the same version but as a GA version (e.g., `x.n.0`). 3. **Patch Versions**: - - For any patch versions, changes will be merged (cherry-picked from main) into the release branch. - - Start a new release automation process for the patch version (e.g., `x.n.p`). + - For any patch versions, changes will be merged (cherry-picked from main) into the release branch. + - Start a new release automation process for the patch version (e.g., `x.n.p`). ```mermaid graph TD @@ -121,7 +121,6 @@ graph TD D1 --> E1[Merge Patches from Main to Release Branch] E1 --> F1[Trigger Release Automation Workflow for x.n.p] F1 --> G1[Patch Version Released] - ``` ## Conclusion diff --git a/server/README.md b/server/README.md index 281ac332f..7127e5c8b 100644 --- a/server/README.md +++ b/server/README.md @@ -7,8 +7,8 @@ 1. [Configuration](#configuration) 1. [Metrics](#metrics) 1. [Design](#design) - 1. [Block Persistence](#block-persistence) - 1. [Bi-directional Producer/Consumer Streaming with gRPC](#bi-directional-producerconsumer-streaming-with-grpc) + 1. [Block Persistence](#block-persistence) + 1. [Bi-directional Producer/Consumer Streaming with gRPC](#bi-directional-producerconsumer-streaming-with-grpc) ## Overview diff --git a/server/docs/configuration.md b/server/docs/configuration.md index 39d4b97af..f046d4e25 100644 --- a/server/docs/configuration.md +++ b/server/docs/configuration.md @@ -9,11 +9,10 @@ The default configuration allows users to quickly get up and running without hav ease of use at the trade-off of some insecure default configuration. Most configuration settings have appropriate defaults and can be left unchanged. It is recommended to browse the properties below and adjust to your needs. - -| Environment Variable | Description | Default Value | -|-----------------------------------|---------------------------------------------------------------------------------------------------------------|---------------| -| PERSISTENCE_STORAGE_ROOT_PATH | The root path for the storage, if not provided will attempt to create a `data` on the working dir of the app. | | -| CONSUMER_TIMEOUT_THRESHOLD_MILLIS | Time to wait for subscribers before disconnecting in milliseconds | 1500 | -| SERVICE_DELAY_MILLIS | Service shutdown delay in milliseconds | 500 | -| MEDIATOR_RING_BUFFER_SIZE | Size of the ring buffer used by the mediator (must be a power of 2) | 67108864 | -| NOTIFIER_RING_BUFFER_SIZE | Size of the ring buffer used by the notifier (must be a power of 2) | 2048 | +| Environment Variable | Description | Default Value | +|:---|:--|---:| +| PERSISTENCE_STORAGE_ROOT_PATH | The root path for the storage, if not provided will attempt to create a `data` on the working dir of the app. | | +| CONSUMER_TIMEOUT_THRESHOLD_MILLIS | Time to wait for subscribers before disconnecting in milliseconds | 1500 | +| SERVICE_DELAY_MILLIS | Service shutdown delay in milliseconds | 500 | +| MEDIATOR_RING_BUFFER_SIZE | Size of the ring buffer used by the mediator (must be a power of 2) | 67108864 | +| NOTIFIER_RING_BUFFER_SIZE | Size of the ring buffer used by the notifier (must be a power of 2) | 2048 | diff --git a/server/docs/design/bidi-producer-consumers-streaming.md b/server/docs/design/bidi-producer-consumers-streaming.md index 136c680e8..326512194 100644 --- a/server/docs/design/bidi-producer-consumers-streaming.md +++ b/server/docs/design/bidi-producer-consumers-streaming.md @@ -7,20 +7,20 @@ 1. [Terms](#terms) 1. [Block Node gRPC Streaming Services API](#block-node-grpc-streaming-services-api) 1. [Approaches](#approaches) - 1. [Approach 1: Directly passing BlockItems from `ProducerBlockItemObserver` to N `ConsumerBlockItemObserver`s](#approach-1-directly-passing-blockitems-from-producerblockitemobserver-to-n-consumerblockitemobservers) - 1. [Approach 2: Use a shared data structure between `ProducerBlockItemObserver` and `ConsumerBlockItemObserver`s. Consumers busy-wait for new BlockItems](#approach-2-use-a-shared-data-structure-between-producerblockitemobserver-and-consumerblockitemobservers-consumers-busy-wait-for-new-blockitems) - 1. [Approach 3: Use a shared data structure between `ProducerBlockItemObserver` and `ConsumerBlockItemObserver`s. Use downstream consumer BlockItemResponses to drive the process of sending new BlockItems](#approach-3-use-a-shared-data-structure-between-producerblockitemobserver-and-consumerblockitemobservers-use-downstream-consumer-blockitemresponses-to-drive-the-process-of-sending-new-blockitems) - 1. [Approach 4: Shared data structure between producer and consumer services. Leveraging the LMAX Disruptor library to manage inter-process pub/sub message-passing between producer and consumers via RingBuffer](#approach-4-shared-data-structure-between-producer-and-consumer-services-leveraging-the-lmax-disruptor-library-to-manage-inter-process-pubsub-message-passing-between-producer-and-consumers-via-ringbuffer) + 1. [Approach 1: Directly passing BlockItems from `ProducerBlockItemObserver` to N `ConsumerBlockItemObserver`s](#approach-1-directly-passing-blockitems-from-producerblockitemobserver-to-n-consumerblockitemobservers) + 1. [Approach 2: Use a shared data structure between `ProducerBlockItemObserver` and `ConsumerBlockItemObserver`s. Consumers busy-wait for new BlockItems](#approach-2-use-a-shared-data-structure-between-producerblockitemobserver-and-consumerblockitemobservers-consumers-busy-wait-for-new-blockitems) + 1. [Approach 3: Use a shared data structure between `ProducerBlockItemObserver` and `ConsumerBlockItemObserver`s. Use downstream consumer BlockItemResponses to drive the process of sending new BlockItems](#approach-3-use-a-shared-data-structure-between-producerblockitemobserver-and-consumerblockitemobservers-use-downstream-consumer-blockitemresponses-to-drive-the-process-of-sending-new-blockitems) + 1. [Approach 4: Shared data structure between producer and consumer services. Leveraging the LMAX Disruptor library to manage inter-process pub/sub message-passing between producer and consumers via RingBuffer](#approach-4-shared-data-structure-between-producer-and-consumer-services-leveraging-the-lmax-disruptor-library-to-manage-inter-process-pubsub-message-passing-between-producer-and-consumers-via-ringbuffer) 1. [Design](#design) - 1. [Producer Registration Flow](#producer-registration-flow) - 1. [Consumer Registration Flow](#consumer-registration-flow) - 1. [Runtime Streaming](#runtime-streaming) - 1. [Entities](#entities) + 1. [Producer Registration Flow](#producer-registration-flow) + 1. [Consumer Registration Flow](#consumer-registration-flow) + 1. [Runtime Streaming](#runtime-streaming) + 1. [Entities](#entities) 1. [Diagrams](#diagrams) - 1. [Producer Registration Flow](#producer-registration-flow-1) - 1. [Consumer Registration Flow](#consumer-registration-flow-1) - 1. [Class Diagram of all Entities and their Relationships](#class-diagram-of-all-entities-and-their-relationships) - 1. [Runtime Stream of BlockItems from Producer to Consumers](#runtime-stream-of-blockitems-from-producer-to-consumers) + 1. [Producer Registration Flow](#producer-registration-flow-1) + 1. [Consumer Registration Flow](#consumer-registration-flow-1) + 1. [Class Diagram of all Entities and their Relationships](#class-diagram-of-all-entities-and-their-relationships) + 1. [Runtime Stream of BlockItems from Producer to Consumers](#runtime-stream-of-blockitems-from-producer-to-consumers) --- @@ -41,8 +41,8 @@ point for custom logic is an implementation of `GrpcService`. ## Goals 1) Consumers must be able to dynamically subscribe and unsubscribe from the live stream of BlockItems emitted by the - producer. When a consumer subscribes to the stream, they will begin receiving BlockItems at the start of the next Block. - BlockItems transiting before the start of the next Block will not be sent to that downstream consumer. + producer. When a consumer subscribes to the stream, they will begin receiving BlockItems at the start of the next Block. + BlockItems transiting before the start of the next Block will not be sent to that downstream consumer. 2) Correct, in-order streaming delivery of BlockItems from a producer to all registered consumers. 3) Minimize latency between the producer and consumers. 4) Minimize CPU resources consumed by the producer and consumers. @@ -112,8 +112,8 @@ Live Streaming (see [BlockNode Issue 21](https://github.com/hashgraph/hedera-blo Drawbacks: 1) Each `ProducerBlockItemObserver` must iterate over the list of subscribed consumers to pass the BlockItem to each `ConsumerBlockItemObserver` before saving the BlockItem to disk and issuing a BlockItemResponse back to the producer. - The linear scaling of consumers will aggregate latency resulting in the last consumer in the list to be penalized - with the sum of the latencies of all consumers before it. + The linear scaling of consumers will aggregate latency resulting in the last consumer in the list to be penalized + with the sum of the latencies of all consumers before it. 2) Dynamically subscribing/unsubscribing `ConsumerBlockItemObserver`s while deterministically broadcasting BlockItems to each consumer in the correct order complicates and slows down the process. It requires thread-safe data structures and synchronization on all reads and writes to ensure new/removed subscribers do not disrupt the @@ -181,7 +181,7 @@ Advantages: Drawbacks: 1) The Disruptor library is a third-party library requiring ramp-up time and integration effort to use it correctly and - effectively. + effectively. 2) Leveraging the Disruptor library requires the communication between the `ProducerBlockItemObserver` and `ConsumerBlockItemObserver`s to be affiliated by subscribing/unsubscribing the downstream consumers to receive the latest BlockItems from the producer via the Disruptor RingBuffer. The process of managing these subscriptions to diff --git a/server/docs/metrics.md b/server/docs/metrics.md index ca71aa28d..41021a680 100644 --- a/server/docs/metrics.md +++ b/server/docs/metrics.md @@ -25,11 +25,11 @@ The purpose of metrics is to provide a way to measure the performance of the sys Prefix: prometheus, ie. `prometheus.configKey` -| ConfigKey | Description | Default | -|----------------------------|---------------------------------------------------------------------------------------|-------------| -| enableEndpoint | either `true` or `false`. Enables or disables the endpoint for metrics | true | -| endpointPortNumber | Port of the Prometheus endpoint | 9999 | -| endpointMaxBacklogAllowed | The maximum number of incoming TCP connections which the system will queue internally | 1 | +| ConfigKey | Description | Default | +|:---|:---|---:| +| enableEndpoint | either `true` or `false`. Enables or disables the endpoint for metrics | true | +| endpointPortNumber | Port of the Prometheus endpoint | 9999 | +| endpointMaxBacklogAllowed | The maximum number of incoming TCP connections which the system will queue internally | 1 | ## Usage @@ -54,9 +54,9 @@ If needed to create another dashboard is possible to include it by adding it to All metrics have `hedera_block_node` prefix. -| Metric Name | Description | Type | -|---------------------------------|---------------------------------------|-----------| -| live_block_items | The number of block items received | Counter | -| blocks_persisted | the number of blocks persisted | Counter | -| subscribers | The number of subscribers | Gauge | -| single_blocks_retrieved | the number of single blocks requested | Counter | +| Metric Name | Description | Type | +|:---|:---|---:| +| live_block_items | The number of block items received | Counter | +| blocks_persisted | the number of blocks persisted | Counter | +| subscribers | The number of subscribers | Gauge | +| single_blocks_retrieved | the number of single blocks requested | Counter | diff --git a/server/docs/quickstart.md b/server/docs/quickstart.md index c042a1261..4682a47a4 100644 --- a/server/docs/quickstart.md +++ b/server/docs/quickstart.md @@ -33,34 +33,34 @@ Refer to the [Configuration](configuration.md) for configuration options. > ``` 1. To quickly build the Server sources (without running tests), do the following: - ```bash - ./gradlew -p server clean build -x test - ``` + ```bash + ./gradlew -p server clean build -x test + ``` 1. To build the Server docker image, do the following: - ```bash - ./gradlew :server:createDockerImage - ``` + ```bash + ./gradlew :server:createDockerImage + ``` ### Run the Server 1. To start the Server, do the following: - ```bash - ./gradlew :server:startDockerContainer - ``` + ```bash + ./gradlew :server:startDockerContainer + ``` ### Run the Server with Debug 1. To start the Server with debug enabled, do the following: - ```bash - ./gradlew :server:startDockerDebugContainer - ``` + ```bash + ./gradlew :server:startDockerDebugContainer + ``` 1. Attach your remote jvm debugger to port 5005. ### Stop the Server 1. To stop the Server do the following: - ```bash - ./gradlew :server:stopDockerContainer - ``` + ```bash + ./gradlew :server:stopDockerContainer + ``` diff --git a/simulator/README.md b/simulator/README.md index e2e64052b..e9c34c577 100644 --- a/simulator/README.md +++ b/simulator/README.md @@ -17,17 +17,18 @@ Uses Dagger2 for dependency injection, the project has a modular structure and d ```plaintext src/java/com/hedera/block/simulator/BlockStreamSimulatorInjectionModule.java ``` + Entry point for the project is `BlockStreamSimulator.java`, in wich the main method is located and has 2 functions: 1. Create/Load the Application Configuration, it does this using Hedera Platform Configuration API. -2. Create a DaggerComponent and instantiate the BlockStreamSimulatorApp class using the DaggerComponent and it registered dependencies. -3. Start the BlockStreamSimulatorApp, contains the orchestration of the different parts of the simulation using generic interfaces and handles the rate of streaming and the exit conditions. +1. Create a DaggerComponent and instantiate the BlockStreamSimulatorApp class using the DaggerComponent and it registered dependencies. +1. Start the BlockStreamSimulatorApp, contains the orchestration of the different parts of the simulation using generic interfaces and handles the rate of streaming and the exit conditions. The BlockStreamSimulatorApp consumes other services that are injected using DaggerComponent, these are: 1. **generator:** responsible for generating blocks, exposes a single interface `BlockStreamManager` and several implementations 1. BlockAsDirBlockStreamManager: generates blocks from a directory, each folder is a block, and block-items are single 'blk' or 'blk.gz' files. - 2. BlockAsFileBlockStreamManager: generates blocks from a single file, each file is a block, used to the format of the CN recordings. (since it loads blocks on memory it can stream really fast, really useful for simple streaming tests) - 3. BlockAsFileLargeDataSets: similar to BlockAsFileBLockStreamManager, but designed to work with GB folders with thousands of big blocks (since it has a high size block and volume of blocks, is useful for performace, load and stress testing) -2. **grpc:** responsible for the communication with the Block-Node, currently only has 1 interface `PublishStreamGrpcClient` and 1 Implementation, however also exposes a `PublishStreamObserver' + 1. BlockAsFileBlockStreamManager: generates blocks from a single file, each file is a block, used to the format of the CN recordings. (since it loads blocks on memory it can stream really fast, really useful for simple streaming tests) + 1. BlockAsFileLargeDataSets: similar to BlockAsFileBLockStreamManager, but designed to work with GB folders with thousands of big blocks (since it has a high size block and volume of blocks, is useful for performace, load and stress testing) +1. **grpc:** responsible for the communication with the Block-Node, currently only has 1 interface `PublishStreamGrpcClient` and 1 Implementation, however also exposes a `PublishStreamObserver' ## Configuration diff --git a/simulator/docs/configuration.md b/simulator/docs/configuration.md index f05bf5b2f..dfa78fbb5 100644 --- a/simulator/docs/configuration.md +++ b/simulator/docs/configuration.md @@ -6,30 +6,34 @@ There are 3 configuration sets: 1. [GrpcConfig](#grpcconfig): contains the configuration for the gRPC communication with the Block-Node. ## BlockStreamConfig + Uses the prefix `blockStream` so all properties should start with `blockStream.` -| Key | Description | Default Value | -|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------| -| `delayBetweenBlockItems` | The delay between each block item in nanoseconds, only applicable when streamingMode=CONSTANT_RATE | `1_500_000` | -| `maxBlockItemsToStream` | exit condition for the simulator and the circular implementations such as `BlockAsDir` or `BlockAsFile` implementations | `10_000` | -| `streamingMode` | can either be `CONSTANT_RATE` or `MILLIS_PER_BLOCK` | `CONSTANT_RATE` | -| `millisecondsPerBlock` | if streamingMode is `MILLIS_PER_BLOCK` this will be the time to wait between blocks in milliseconds | `1_000` | -| `blockItemsBatchSize` | the number of block items to send in a single batch, however if a block has less block items, it will send all the items in a block | `1_000` | +| Key | Description | Default Value | +|:---|:---|---:| +| `delayBetweenBlockItems` | The delay between each block item in nanoseconds, only applicable when streamingMode=CONSTANT_RATE | `1_500_000` | +| `maxBlockItemsToStream` | exit condition for the simulator and the circular implementations such as `BlockAsDir` or `BlockAsFile` implementations | `10_000` | +| `streamingMode` | can either be `CONSTANT_RATE` or `MILLIS_PER_BLOCK` | `CONSTANT_RATE` | +| `millisecondsPerBlock` | if streamingMode is `MILLIS_PER_BLOCK` this will be the time to wait between blocks in milliseconds | `1_000` | +| `blockItemsBatchSize` | the number of block items to send in a single batch, however if a block has less block items, it will send all the items in a block | `1_000` | ## GeneratorConfig + Uses the prefix `generator` so all properties should start with `generator.` -| Key | Description | Default Value | -|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------| -| `generationMode` | The desired generation Mode to use, it can only be `DIR` or `AD_HOC` | `DIR` | -| `folderRootPath` | If the generationMode is DIR this will be used as the source of the recording to stream to the Block-Node | `` | -| `managerImplementation` | The desired implementation of the BlockStreamManager to use, it can only be `BlockAsDirBlockStreamManager`, `BlockAsFileBlockStreamManager` or `BlockAsFileLargeDataSets` | `BlockAsFileBlockStreamManager` | -| `paddedLength` | on the `BlockAsFileLargeDataSets` implementation, the length of the padded left zeroes `000001.blk.gz` | 36 | -| `fileExtension` | on the `BlockAsFileLargeDataSets` implementation, the extension of the files to be streamed | `.blk.gz` | + +| Key | Description | Default Value | +|:---|:---|---:| +| `generationMode` | The desired generation Mode to use, it can only be `DIR` or `AD_HOC` | `DIR` | +| `folderRootPath` | If the generationMode is DIR this will be used as the source of the recording to stream to the Block-Node | `` | +| `managerImplementation` | The desired implementation of the BlockStreamManager to use, it can only be `BlockAsDirBlockStreamManager`, `BlockAsFileBlockStreamManager` or `BlockAsFileLargeDataSets` | `BlockAsFileBlockStreamManager` | +| `paddedLength` | on the `BlockAsFileLargeDataSets` implementation, the length of the padded left zeroes `000001.blk.gz` | 36 | +| `fileExtension` | on the `BlockAsFileLargeDataSets` implementation, the extension of the files to be streamed | `.blk.gz` | ## GrpcConfig + Uses the prefix `grpc` so all properties should start with `grpc.` -| Key | Description | Default Value | -|-----------------|----------------------------|---------------| -| `serverAddress` | The host of the Block-Node | `localhost` | -| `port` | The port of the Block-Node | `8080` | +| Key | Description | Default Value | +|:---|:---|---:| +| `serverAddress` | The host of the Block-Node | `localhost` | +| `port` | The port of the Block-Node | `8080` | diff --git a/simulator/docs/quickstart.md b/simulator/docs/quickstart.md index 0a5aff3a5..9022f548f 100644 --- a/simulator/docs/quickstart.md +++ b/simulator/docs/quickstart.md @@ -33,9 +33,9 @@ Refer to the [Configuration](configuration.md) for configuration options. > ``` 1. To quickly build the Simulator sources (without running tests), do the following: - ```bash - ./gradlew -p simulator clean build -x test - ``` + ```bash + ./gradlew -p simulator clean build -x test + ``` ### Run the Server first @@ -46,15 +46,15 @@ get started with the application. ### Run the Simulator 1. To start the Simulator, do the following: - ```bash - ./gradlew :simulator:run - ``` + ```bash + ./gradlew :simulator:run + ``` ### Run the Simulator with Debug 1. To start the Simulator with debug enabled, do the following: - ```bash - ./gradlew :simulator:run --debug-jvm - ``` + ```bash + ./gradlew :simulator:run --debug-jvm + ``` 1. Attach your remote jvm debugger to port 5005. diff --git a/tools/README.md b/tools/README.md index 10658f7e8..d479002fc 100644 --- a/tools/README.md +++ b/tools/README.md @@ -27,13 +27,16 @@ Converts a binary block stream to JSON `Usage: subcommands json [-t] [-ms=] [...]` **Options:** + - `-ms ` or `--min-size=` - - Filter to only files bigger than this minimum file size in megabytes + - Filter to only files bigger than this minimum file size in megabytes + - `-t` or `--transactions` - - expand transactions, this is no longer pure json conversion but is very useful making the + - expand transactions, this is no longer pure json conversion but is very useful making the transactions human-readable. + - `...` - - The block files or directories of block files to convert to JSON + - The block files or directories of block files to convert to JSON ### The `info` Subcommand @@ -42,14 +45,18 @@ Prints info for block files `Usage: subcommands info [-c] [-ms=] [-o=] [...]` **Options:** + - `-c` or `--csv` - - Enable CSV output mode (default: false) + - Enable CSV output mode (default: false) + - `-ms ` or `--min-size=` - - Filter to only files bigger than this minimum file size in megabytes + - Filter to only files bigger than this minimum file size in megabytes + - `-o ` or `--output-file=` - - Output to file rather than stdout + - Output to file rather than stdout + - `...` - - The block files or directories of block files to print info for + - The block files or directories of block files to print info for ## Running from command line diff --git a/tools/docs/quickstart.md b/tools/docs/quickstart.md index 1a5ffdea0..b327ee67e 100644 --- a/tools/docs/quickstart.md +++ b/tools/docs/quickstart.md @@ -26,16 +26,16 @@ > ``` 1. To quickly build the Tools sources (without running tests), do the following: - ```bash - ./gradlew -p tools clean build -x test - ``` + ```bash + ./gradlew -p tools clean build -x test + ``` ### Run the Tools 1. To run the Tools, do the following: - ```bash - # Here is an example of running the info command with the help option, simply - # replace `info --help` with the desired command and options to run the tools - # quickly using the `./gradlew run` task. - ./gradlew -q :tools:run --args="info --help" - ``` + ```bash + # Here is an example of running the info command with the help option, simply + # replace `info --help` with the desired command and options to run the tools + # quickly using the `./gradlew run` task. + ./gradlew -q :tools:run --args="info --help" + ``` From f587a14a006f2792645bda5223aa8651472bf9b6 Mon Sep 17 00:00:00 2001 From: Atanas Atanasov Date: Thu, 7 Nov 2024 13:30:25 +0200 Subject: [PATCH 10/11] typo Signed-off-by: Atanas Atanasov --- server/docs/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server/docs/configuration.md b/server/docs/configuration.md index f046d4e25..50f2cd6d9 100644 --- a/server/docs/configuration.md +++ b/server/docs/configuration.md @@ -10,7 +10,7 @@ ease of use at the trade-off of some insecure default configuration. Most config defaults and can be left unchanged. It is recommended to browse the properties below and adjust to your needs. | Environment Variable | Description | Default Value | -|:---|:--|---:| +|:---|:---|---:| | PERSISTENCE_STORAGE_ROOT_PATH | The root path for the storage, if not provided will attempt to create a `data` on the working dir of the app. | | | CONSUMER_TIMEOUT_THRESHOLD_MILLIS | Time to wait for subscribers before disconnecting in milliseconds | 1500 | | SERVICE_DELAY_MILLIS | Service shutdown delay in milliseconds | 500 | From 3324c4d8158aef8d834c6c20508186029dafb55c Mon Sep 17 00:00:00 2001 From: Atanas Atanasov Date: Thu, 7 Nov 2024 13:55:30 +0200 Subject: [PATCH 11/11] some nitpicks Signed-off-by: Atanas Atanasov --- README.md | 5 ++--- docs/{README.md => overview.md} | 5 +++-- 2 files changed, 5 insertions(+), 5 deletions(-) rename docs/{README.md => overview.md} (84%) diff --git a/README.md b/README.md index d4bd0f3f8..4c523951c 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ Implementation of the Hedera Block Node, which is responsible for consuming the [![Latest Version](https://img.shields.io/github/v/tag/hashgraph/hedera-block-node?sort=semver&label=version)](README.md) [![Made With](https://img.shields.io/badge/made_with-java-blue)](https://github.com/hashgraph/hedera-block-node/) -[![Development Branch](https://img.shields.io/badge/docs-quickstart-green.svg)](docs/README.md) +[![Development Branch](https://img.shields.io/badge/docs-quickstart-green.svg)](docs/overview.md) [![License](https://img.shields.io/badge/license-apache2-blue.svg)](LICENSE) ## Prerequisites @@ -40,10 +40,9 @@ Implementation of the Hedera Block Node, which is responsible for consuming the - `suites`: A set of e2e tests that can be used to verify the correctness of the block node. - `tools`: A set of command line tools for working with block stream files. - ## Getting Started -Refer to the [Hedera Block Node Documentation Overview](docs/README.md) for more information about the project, design and guides. +Refer to the [Hedera Block Node Documentation Overview](docs/overview.md) for more information about the project, design and guides. ## Support diff --git a/docs/README.md b/docs/overview.md similarity index 84% rename from docs/README.md rename to docs/overview.md index f39c90b19..018e76e01 100644 --- a/docs/README.md +++ b/docs/overview.md @@ -4,14 +4,15 @@ This document will outline the structure and content of Block Node documentation ## Table of Contents -1. [Introduction](#introduction) +1. [Projects](#projects) 1. [Concepts](#concepts) 1. [Release](#release) -## Introduction +## Projects - [Hedera Block Node](../server/README.md): Provides an introduction to Hedera Block Node, its purpose and how it can be used. - [Simulator](../simulator/README.md): Offers instructions on how the Simulator can be utilized. +- [Tools](../tools/README.md): Provides information on the command line tools available. ## Concepts