Skip to content

Commit

Permalink
Cleanup README (#612)
Browse files Browse the repository at this point in the history
  • Loading branch information
@chia7712
chia7712 authored Aug 23, 2022
1 parent 8172942 commit 41d8181
Showing 1 changed file with 10 additions and 156 deletions.
166 changes: 10 additions & 156 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,162 +13,16 @@

# Kafka Tools

This project offers many kafka tools to simplify the life for kafka users.

1. [Kafka quick start](#kafka-cluster-quick-start): set up a true kafka cluster in one minute
2. [Kafka performance](./docs/performance_benchmark.md): 可產生不同類型資料集測試讀寫速度及E2E延遲的工具
3. [Kafka_Prometheus](./docs/run_prometheus.md): 整合 Kafka 與 Prometheus
4. [快速啟動Grafana](./docs/run_grafana.md): 建置圖形化介面監控Kafka server、Host端資源使用量
5. [Kafka metric explorer](#kafka-metric-explorer): utility for accessing kafka Mbean metrics via JMX.
6. [Kafka replica syncing monitor](#Kafka-replica-syncing-monitor): Tracking replica syncing progress.
7. [Astraea Web Server 中文文件連結](./docs/web_server/README.md)

[Github packages](https://github.com/orgs/skiptests/packages?repo_name=astraea) offers the docker image to run mentioned tools
```shell
./docker/start_app.sh web --bootstrap.servers 192.168.50.178:19993 --port 12345"
```
---
## Kafka Cluster Quick Start
The following scripts can build a kafka cluster by containers in one minute.
### Run Zookeeper
```shell
./docker/start_zookeeper.sh
```
The script creates a zookeeper instance by container. Also, it will show the command used to add broker instance. For example:
```shell
=================================================
run ./docker/start_broker.sh zookeeper.connect=192.168.50.178:17228 to join kafka broker
=================================================
```
You can define `VERSION` to change the binary version.
### Run Kafka Broker
After the zk env is running, you can copy the command (see above example) from zk script output to set up kafka. For example:
```shell
./docker/start_broker.sh zookeeper.connect=192.168.50.178:17228
```
The console will show the broker connection information and JMX address. For example:
```shell
=================================================
broker id: 677
broker address: 192.168.50.178:12747
jmx address: 192.168.50.178:10216
exporter address: 192.168.50.178:10558
=================================================
```
1. `broker address` is used by kafka client code. The alias is bootstrap server.
2. `jmx address` exports the java metrics by JMX
3. `exporter address` is the address of prometheus exporter.
You can set `CONFLUENT_BROKER` to true, if you want start the confluent version of the kafka cluster. For example:
```shell
env CONFLUENT_BROKER=true ./docker/start_broker.sh zookeeper.connect=192.168.50.178:17228
```
The console will show the broker connection information and exporter address. For example:
```shell
=================================================
broker id: 1001
broker address: 192.168.103.39:15230
exporter address: 192.168.103.39:18928
=================================================
```
There are 4 useful ENVs which can change JVM/container configuration.
1. VERSION -> define the kafka version
2. REVISION -> define the revision of kafka source code. If this is defined, it will run distribution based on the source code
3. HEAP_OPTS -> define JVM memory options
4. DATA_FOLDERS -> define the host folders used by broker. You should define it if you want to keep data after terminating container
### Run Node Exporter
[Node Exporter](https://github.com/prometheus/node_exporter) is a famous utility for exporting machine metrics. It is
recommended using node exporter in conjunction with Prometheus to observe the test environment state.
```shell
./docker/start_node_exporter.sh
```
```shell
[INFO] Container ID of node_exporter: d67d5d1daaaaf57792d145a8a8a5bd470207e698c8ca544f3023bdfcac914271
[INFO] node_exporter running at http://192.168.0.2:9100
```
### Run Grafana
[Grafana](https://github.com/grafana/grafana) is a famous application for display system states. It is recommended to use Grafana
in conjunction with Prometheus to observe the test environment state.
#### Start Grafana
This project offers a way to quickly create a Grafana container instance for **test purpose**.
```shell
./docker/start_grafana.sh start
```
```shell
aa8a47da91a2e0974a38690525f9148c9697f7ffc752611ef06248ffb09ef53a
[INFO] Default username/password for grafana docker image is admin/admin
[INFO] Access Grafana dashboard here: http://192.168.0.2:3000
```
#### Add Prometheus DataSource
Grafana needs to know where the metrics are, so he can show you the pretty diagram. The first step is setting up the
the data source for your Grafana instance.
The following command set up a Prometheus data source for the Grafana instance we previously created.
```shell
./docker/start_grafana.sh add_prom_source <USERNAME>:<PASSWORD> Prometheus http://192.168.0.2:9090
```
```json
{
"datasource": {
"id": 1,
"uid": "7jbIw-Tnz",
"orgId": 1,
"name": "Prometheus",
"type": "prometheus",
"typeLogoUrl": "",
"access": "proxy",
"url": "http://192.168.0.2:9090",
"password": "",
"user": "",
"database": "",
"basicAuth": false,
"basicAuthUser": "",
"basicAuthPassword": "",
"withCredentials": false,
"isDefault": false,
"jsonData": {},
"secureJsonFields": {},
"version": 1,
"readOnly": false
},
"id": 1,
"message": "Datasource added",
"name": "Prometheus"
}
```
`Astraea` 提供各式工具來降低`Kafka`門檻以及提高`Kafka`效能

1. [快速啟動 Zookeeper ](./docs/run_zookeeper.md): 使用容器化的方式快速建立`zookeeper`服務
2. [快速啟動 Kafka ](./docs/run_kafka_broker.md): 使用容器化的方式快速建立`kafka broker`服務
3. [Performance Tool ](./docs/performance_benchmark.md): 可模擬多種使用情境來驗證`Kafka`叢集的吞吐量和延遲
4. [快速啟動 Prometheus ](./docs/run_prometheus.md): 建構`Kafka`叢集資訊收集系統
5. [快速啟動 Grafana ](./docs/run_grafana.md): 建置圖形化介面監控`kafka`叢集使用狀況
6. [Web Server](./docs/web_server/README.md): 可透過`Restful APIs`操作`Kafka`叢集
7. [Kafka metric explorer](#kafka-metric-explorer): utility for accessing kafka Mbean metrics via JMX. (deprecated)
8. [Kafka replica syncing monitor](#Kafka-replica-syncing-monitor): Tracking replica syncing progress. (deprecated)
---

## Kafka Metric Explorer
Expand Down

0 comments on commit 41d8181

Please sign in to comment.