diff --git a/README.md b/README.md index 12060c829d..646dd599b2 100644 --- a/README.md +++ b/README.md @@ -13,162 +13,16 @@ # Kafka Tools -This project offers many kafka tools to simplify the life for kafka users. - -1. [Kafka quick start](#kafka-cluster-quick-start): set up a true kafka cluster in one minute -2. [Kafka performance](./docs/performance_benchmark.md): 可產生不同類型資料集測試讀寫速度及E2E延遲的工具 -3. [Kafka_Prometheus](./docs/run_prometheus.md): 整合 Kafka 與 Prometheus -4. [快速啟動Grafana](./docs/run_grafana.md): 建置圖形化介面監控Kafka server、Host端資源使用量 -5. [Kafka metric explorer](#kafka-metric-explorer): utility for accessing kafka Mbean metrics via JMX. -6. [Kafka replica syncing monitor](#Kafka-replica-syncing-monitor): Tracking replica syncing progress. -7. [Astraea Web Server 中文文件連結](./docs/web_server/README.md) - -[Github packages](https://github.com/orgs/skiptests/packages?repo_name=astraea) offers the docker image to run mentioned tools -```shell -./docker/start_app.sh web --bootstrap.servers 192.168.50.178:19993 --port 12345" -``` - ---- - -## Kafka Cluster Quick Start - -The following scripts can build a kafka cluster by containers in one minute. - -### Run Zookeeper - -```shell -./docker/start_zookeeper.sh -``` - -The script creates a zookeeper instance by container. Also, it will show the command used to add broker instance. For example: - -```shell -================================================= -run ./docker/start_broker.sh zookeeper.connect=192.168.50.178:17228 to join kafka broker -================================================= -``` - -You can define `VERSION` to change the binary version. - -### Run Kafka Broker - -After the zk env is running, you can copy the command (see above example) from zk script output to set up kafka. For example: - -```shell -./docker/start_broker.sh zookeeper.connect=192.168.50.178:17228 -``` - -The console will show the broker connection information and JMX address. For example: - -```shell -================================================= -broker id: 677 -broker address: 192.168.50.178:12747 -jmx address: 192.168.50.178:10216 -exporter address: 192.168.50.178:10558 -================================================= -``` - -1. `broker address` is used by kafka client code. The alias is bootstrap server. -2. `jmx address` exports the java metrics by JMX -3. `exporter address` is the address of prometheus exporter. - -You can set `CONFLUENT_BROKER` to true, if you want start the confluent version of the kafka cluster. For example: - -```shell -env CONFLUENT_BROKER=true ./docker/start_broker.sh zookeeper.connect=192.168.50.178:17228 -``` - -The console will show the broker connection information and exporter address. For example: - -```shell -================================================= -broker id: 1001 -broker address: 192.168.103.39:15230 -exporter address: 192.168.103.39:18928 -================================================= -``` - -There are 4 useful ENVs which can change JVM/container configuration. - -1. VERSION -> define the kafka version -2. REVISION -> define the revision of kafka source code. If this is defined, it will run distribution based on the source code -3. HEAP_OPTS -> define JVM memory options -4. DATA_FOLDERS -> define the host folders used by broker. You should define it if you want to keep data after terminating container - -### Run Node Exporter - -[Node Exporter](https://github.com/prometheus/node_exporter) is a famous utility for exporting machine metrics. It is -recommended using node exporter in conjunction with Prometheus to observe the test environment state. - -```shell -./docker/start_node_exporter.sh -``` - -```shell -[INFO] Container ID of node_exporter: d67d5d1daaaaf57792d145a8a8a5bd470207e698c8ca544f3023bdfcac914271 -[INFO] node_exporter running at http://192.168.0.2:9100 -``` - -### Run Grafana - -[Grafana](https://github.com/grafana/grafana) is a famous application for display system states. It is recommended to use Grafana -in conjunction with Prometheus to observe the test environment state. - -#### Start Grafana - -This project offers a way to quickly create a Grafana container instance for **test purpose**. - -```shell -./docker/start_grafana.sh start -``` - -```shell -aa8a47da91a2e0974a38690525f9148c9697f7ffc752611ef06248ffb09ef53a -[INFO] Default username/password for grafana docker image is admin/admin -[INFO] Access Grafana dashboard here: http://192.168.0.2:3000 -``` - -#### Add Prometheus DataSource - -Grafana needs to know where the metrics are, so he can show you the pretty diagram. The first step is setting up the -the data source for your Grafana instance. - -The following command set up a Prometheus data source for the Grafana instance we previously created. - -```shell -./docker/start_grafana.sh add_prom_source : Prometheus http://192.168.0.2:9090 -``` -```json -{ - "datasource": { - "id": 1, - "uid": "7jbIw-Tnz", - "orgId": 1, - "name": "Prometheus", - "type": "prometheus", - "typeLogoUrl": "", - "access": "proxy", - "url": "http://192.168.0.2:9090", - "password": "", - "user": "", - "database": "", - "basicAuth": false, - "basicAuthUser": "", - "basicAuthPassword": "", - "withCredentials": false, - "isDefault": false, - "jsonData": {}, - "secureJsonFields": {}, - "version": 1, - "readOnly": false - }, - "id": 1, - "message": "Datasource added", - "name": "Prometheus" -} -``` - +`Astraea` 提供各式工具來降低`Kafka`門檻以及提高`Kafka`效能 + +1. [快速啟動 Zookeeper ](./docs/run_zookeeper.md): 使用容器化的方式快速建立`zookeeper`服務 +2. [快速啟動 Kafka ](./docs/run_kafka_broker.md): 使用容器化的方式快速建立`kafka broker`服務 +3. [Performance Tool ](./docs/performance_benchmark.md): 可模擬多種使用情境來驗證`Kafka`叢集的吞吐量和延遲 +4. [快速啟動 Prometheus ](./docs/run_prometheus.md): 建構`Kafka`叢集資訊收集系統 +5. [快速啟動 Grafana ](./docs/run_grafana.md): 建置圖形化介面監控`kafka`叢集使用狀況 +6. [Web Server](./docs/web_server/README.md): 可透過`Restful APIs`操作`Kafka`叢集 +7. [Kafka metric explorer](#kafka-metric-explorer): utility for accessing kafka Mbean metrics via JMX. (deprecated) +8. [Kafka replica syncing monitor](#Kafka-replica-syncing-monitor): Tracking replica syncing progress. (deprecated) --- ## Kafka Metric Explorer