diff --git a/README.md b/README.md index 15ae4db61..9b8cdcf6a 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ The best demo to start with is [cp-demo](https://github.com/confluentinc/cp-demo | Demo | Local | Docker | Description | ------------------------------------------ | ----- | ------ | --------------------------------------------------------------------------- -| [Clickstream](clickstream/README.md) | [Y](clickstream/README.md) | [Y](https://docs.confluent.io/current/ksql/docs/tutorials/clickstream-docker.html#ksql-clickstream-docker?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | Automated version of the [KSQL clickstream demo](https://docs.confluent.io/current/ksql/docs/tutorials/clickstream-docker.html#ksql-clickstream-docker?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top)
+| [Clickstream](clickstream/README.md) | N | [Y](https://docs.confluent.io/current/ksql/docs/tutorials/clickstream-docker.html#ksql-clickstream-docker?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | Automated version of the [KSQL clickstream demo](https://docs.confluent.io/current/ksql/docs/tutorials/clickstream-docker.html#ksql-clickstream-docker?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top)
| [Kafka Tutorials](https://kafka-tutorials.confluent.io?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | [Y](https://kafka-tutorials.confluent.io?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | [Y](https://kafka-tutorials.confluent.io?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | Collection of common event streaming use cases, with each tutorial featuring an example scenario and several complete code solutions
| [KSQL UDF](https://github.com/confluentinc/demo-scene/blob/master/ksql-udf-advanced-example/README.md?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | [Y](https://github.com/confluentinc/demo-scene/blob/master/ksql-udf-advanced-example/README.md?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) | N | Advanced [KSQL User-Defined Function (UDF)](https://www.confluent.io/blog/build-udf-udaf-ksql-5-0?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.top) use case for connected cars
| [KSQL workshop](https://github.com/confluentinc/demo-scene/blob/master/ksql-workshop/) | N | [Y](https://github.com/confluentinc/demo-scene/blob/master/ksql-workshop/) | showcases Kafka event stream processing using KSQL and can run self-guided as a KSQL workshop
diff --git a/clickstream/README.md b/clickstream/README.md index 78294c529..b62d7f07c 100644 --- a/clickstream/README.md +++ b/clickstream/README.md @@ -1,29 +1,10 @@ ![image](../images/confluent-logo-300-2.png) -## Overview +# Overview -This demo shows how KSQL can be used to process a stream of click data, aggregate and filter it, and join to information about the users. Visualisation of the results is provided by Grafana, on top of data streamed to Elasticsearch. +This demo shows how KSQL can be used to process a stream of click data, aggregate and filter it, and join to information about the users. +Visualisation of the results is provided by Grafana, on top of data streamed to Elasticsearch. -You can [run it using Docker](https://docs.confluent.io/current/ksql/docs/tutorials/clickstream-docker.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clickstream), or with a local install of Confluent Platform and following the instructions below. +# Documentation -## Prerequisites - -* [Common demo prerequisites](https://github.com/confluentinc/examples#prerequisites) -* [Confluent Platform 5.4](https://www.confluent.io/download/?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clickstream) -* `jq` installed on your machine -* [Elasticsearch 5.6.5](https://www.elastic.co/downloads/past-releases/elasticsearch-5-6-5) to export data from Kafka - * If you do not want to use Elasticsearch, comment out ``check_running_elasticsearch`` in the ``start.sh`` script -* Grafana 5.0.3 to visualize data - * If you do not want to use Grafana, comment out ``check_running_grafana`` in the ``start.sh`` script - -Note: this demo has been validated only on macOS. - -## What Should I see? - -After you run `./start.sh`: - -* If you are running Confluent Platform, open your browser and navigate to the Control Center web interface Monitoring -> Data streams tab at http://localhost:9021/monitoring/streams to see throughput and latency performance of the KSQL queries -* If you are running Confluent Platform, use Control Center to view and create KSQL queries. Otherwise, run the KSQL CLI `ksql http://localhost:8088`. -* Navigate to the Grafana dashboard at http://localhost:3000/dashboard/db/click-stream-analysis. Login with user ID `admin` and password `admin`. - -![image](images/clickstream-dashboard.png) +You can find the documentation for running this demo and its accompanying tutorial at [https://docs.confluent.io/current/ksql/docs/tutorials/clickstream-docker.html](https://docs.confluent.io/current/ksql/docs/tutorials/clickstream-docker.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.clickstream) diff --git a/clickstream/docker-compose.yml b/clickstream/docker-compose.yml index 283129bfa..663667c5a 100644 --- a/clickstream/docker-compose.yml +++ b/clickstream/docker-compose.yml @@ -194,16 +194,3 @@ services: GRAFANA_HOST: grafana volumes: - $PWD/ksql/ksql-clickstream-demo/demo:/scripts - -# Kibana's useful for exploring the data, and/or dev tools for -# Elasticsearch. It's not core to the demo though, so not enabled -# by default. - # kibana: - # image: docker.elastic.co/kibana/kibana:6.3.0 - # depends_on: - # - elasticsearch - # ports: - # - 5601:5601 - # environment: - # xpack.security.enabled: "false" - # discovery.type: "single-node" diff --git a/clickstream/start.sh b/clickstream/start.sh index 70e051993..87c0bd8e1 100755 --- a/clickstream/start.sh +++ b/clickstream/start.sh @@ -3,30 +3,19 @@ # Source library . ../utils/helper.sh -check_env || exit 1 -check_jq || exit 1 -check_running_cp ${CP_VERSION_MAJOR} || exit 1 -check_running_elasticsearch 5.6.5 || exit 1 -check_running_grafana 5.0.3 || exit 1 - ./stop.sh -cp -nR ksql/ksql-clickstream-demo/demo/connect-config/null-filter-4.0.0-SNAPSHOT.jar $CONFLUENT_HOME/share/java/kafka-connect-elasticsearch/. -confluent local start - -if check_cp; then PROPERTIES=" propertiesFile=$CONFLUENT_HOME/etc/ksql/datagen.properties"; else PROPERTIES=""; fi -ksql-datagen -daemon quickstart=clickstream format=json topic=clickstream maxInterval=100 iterations=500000 $PROPERTIES &>/dev/null & -ksql-datagen quickstart=clickstream_codes format=json topic=clickstream_codes maxInterval=20 iterations=100 $PROPERTIES &>/dev/null & -ksql-datagen quickstart=clickstream_users format=json topic=clickstream_users maxInterval=10 iterations=1000 $PROPERTIES &>/dev/null & -sleep 5 +docker-compose up -d +echo -e "\nSleeping 60 seconds while demo starts\n" +sleep 60 -ksql http://localhost:8088 </dev/null) -(cd ksql/ksql-clickstream-demo/demo/ && ./elastic-dynamic-template.sh &>/dev/null) -(cd ksql/ksql-clickstream-demo/demo/ && ./clickstream-analysis-dashboard.sh &>/dev/null) +docker-compose exec elasticsearch bash -c '/scripts/elastic-dynamic-template.sh' +docker-compose exec kafka-connect bash -c '/scripts/ksql-tables-to-grafana.sh' +docker-compose exec grafana bash -c '/scripts/clickstream-analysis-dashboard.sh' echo -e "\n-> Navigate to the Grafana dashboard at http://localhost:3000/dashboard/db/click-stream-analysis.\n\nLogin with user ID admin and password admin.\n\n" diff --git a/clickstream/stop.sh b/clickstream/stop.sh index e2602b051..891c61f0d 100755 --- a/clickstream/stop.sh +++ b/clickstream/stop.sh @@ -1,11 +1,3 @@ #!/bin/bash -# Source library -. ../utils/helper.sh - -check_env || exit 1 - -jps | grep DataGen | awk '{print $1;}' | xargs kill -9 -confluent local destroy - -# jps | grep Elasticsearch | awk '{print $1;}' | xargs kill -9 +docker-compose down --volumes