Configure Analytics and update examples and documentation (#24)

# What was done - Update readme documentation - Update documentation and examples based on the latest Couchbase documentation - Add missing semi-colons to NodeJS file - Add ability to configure the Analytics service - Add ability to set the cluster name - Add ability to set Eventing service RAM
brantburnett · Dec 22, 2020 · 4a6f620 · 4a6f620
1 parent a640ac2
commit 4a6f620
Show file tree

Hide file tree

Showing 12 changed files with 129 additions and 29 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -26,7 +26,7 @@ COPY ./scripts/ /scripts/
 COPY ./startup/ /startup/
 
 # Configure default environment
-ENV CB_DATARAM=512 CB_INDEXRAM=256 CB_SEARCHRAM=256 \
+ENV CB_DATARAM=512 CB_INDEXRAM=256 CB_SEARCHRAM=256 CB_ANALYTICSRAM=1024 CB_EVENTINGRAM=256 \
 	CB_SERVICES=kv,n1ql,index,fts CB_INDEXSTORAGE=forestdb \
 	CB_USERNAME=Administrator CB_PASSWORD=password \
 	FAKEIT_BUCKETTIMEOUT=5000

diff --git a/README.md b/README.md
@@ -37,17 +37,22 @@ COPY . /startup/
 
 The following environment variables can be set to change the Couchbase Server configuration:
 
-| Env Variable    | Description                                                    |
-| ------------    | -----------                                                    |
-| CB_DATARAM      | Data service RAM in megabytes, default `512`                   |
-| CB_INDEXRAM     | Index service RAM in megabytes, default `256`                  |
-| CB_SEARCHRAM    | Search (FTS) service RAM in megabytes, default `256`           |
-| CB_SERVICES     | Services to enable, default `kv,n1ql,index,fts`                |
-| CB_INDEXSTORAGE | Index storage mode, `forestdb` (default) or `memory_optimized` |
-| CB_USERNAME     | Couchbase user name, default `Administrator`                   |
-| CB_PASSWORD     | Couchbase password, default `password`                         |
-
-Values for CB_SERVICES and CB_INDEXSTORAGE correspond to parameters for the [Couchbase REST API](https://docs.couchbase.com/server/current/rest-api/rest-bucket-create.html).
+| Env Variable    | Description                                                                                |
+| ------------    | -----------                                                                                |
+| CB_CLUSTER_NAME | Specify the name of the cluster                                                            |
+| CB_DATARAM      | Data service RAM in megabytes, default `512`                                               |
+| CB_INDEXRAM     | Index service RAM in megabytes, default `256`                                              |
+| CB_SEARCHRAM    | Search (FTS) service RAM in megabytes, default `256`                                       |
+| CB_ANALYTICSRAM | Analytics service RAM in megabytes. Only applicable if `cbas` is added to `CB_SERVICES`    |
+| CB_EVENTINGRAM  | Eventing service RAM in megabytes. Only applicable if `eventing` is added to `CB_SERVICES` |
+| CB_SERVICES     | Services to enable, default `kv,n1ql,index,fts`                                            |
+| CB_INDEXSTORAGE | Index storage mode, `forestdb` (default) or `memory_optimized`                             |
+| CB_USERNAME     | Couchbase user name, default `Administrator`                                               |
+| CB_PASSWORD     | Couchbase password, default `password`                                                     |
+
+Values for CB_SERVICES and CB_INDEXSTORAGE correspond to parameters for the [Couchbase REST API](https://docs.couchbase.com/server/current/rest-api/rest-node-provisioning.html).
+
+**NOTE:** If you configure `CB_SERVICES` to create the `cbas` analytics service, make sure you set `CB_ANALYTICSRAM` to a minimum of `1024`.
 
 ### Bucket Configuration
 
@@ -78,7 +83,7 @@ To configure your buckets, simply place a `buckets.json` file in the `/startup`
 ]
 ```
 
-Attribute names and values in this file correspond with the [Couchbase REST API create bucket endpoint](https://developer.couchbase.com/documentation/server/4.6/rest-api/rest-bucket-create.html).
+Attribute names and values in this file correspond with the [Couchbase REST API create bucket endpoint](https://docs.couchbase.com/server/current/rest-api/rest-bucket-create.html).
 
 If this file is not overridden in your image, it will create a single bucket named `default` with a RAM quota of 100MB.
 
@@ -161,13 +166,43 @@ Within this file, you can define the `CREATE INDEX` statements for your bucket,
 ```sql
 CREATE PRIMARY INDEX `#primary` ON default WITH {"defer_build": true};
 CREATE INDEX `Types` ON default (`type`) WITH {"defer_build": true};
-BUILD INDEX ON default (`#primary`, `Types`)
+BUILD INDEX ON default (`#primary`, `Types`);
 ```
 
 ### Creating Indexes with YAML
 
 Alternatively, you may add YAML files with index definitions under the `/startup/<bucketname>/indexes` folder.  This operation uses [couchbase-index-manager](https://www.npmjs.com/package/couchbase-index-manager) to create the indexes.  [See here](https://www.npmjs.com/package/couchbase-index-manager#definition-files) for an explanation of the YAML file format.
 
+### Analytics Dataset Setup
+
+To setup the analytics service datasets, add a directory under the `/startup/<bucketname>/analytics` folder. Within this folder create a text file named `dataset.json`. For example, `/startup/default/analytics/dataset.json`.  Note that the names are case sensitive.
+
+Within this file, create a key called `statements` with an array as the value. Within the array you can define your `CREATE DATAVERSE` and `CREATE DATASET` statements for your bucket, separated by semicolons. Below is an example showing what the `dataset.json` structure should look like. **IMPORTANT** Make sure to append the proper `USE` statement to each `CREATE DATASET` statement so that it's placed in the proper DATAVERSE. Additionaly, always end your file with a `CONNECT LINK Local;` statement.
+
+```json
+{
+  "statements": [
+    "CREATE DATAVERSE `sample` IF NOT EXISTS;",
+    "USE `sample`; CREATE DATASET IF NOT EXISTS users ON `sample` WHERE `type` = 'user';",
+    "USE `sample`; CONNECT LINK Local;"
+  ]
+}
+```
+
+### Creating Analytics Indexes
+
+To create analytics indexes, add a directory under the `/startup/<bucketname>/analytics` folder.  Within this folder create a text file named `indexes.json`. For example, `/startup/default/analytics/indexes.json`.  Note that the names are case sensitive.
+
+Within this file, create a key called `statements` with an array as the value. Within the array you can define your `CREATE INDEX` statements for your DATASET, separated by semicolons. **IMPORTANT** Make sure your queries always start with a `USE` statment otherwise the query engine will have no idea which DATAVERSE to associate the index with.
+
+```json
+{
+  "statements": [
+    "USE `sample`; CREATE INDEX `idx_users` IF NOT EXISTS ON `users` (id: string);"
+  ]
+}
+```
+
 ### Creating Full Text Search Indexes
 
 To create FTS indexes, add a directory underneath `/startup` with the name of your bucket, and underneath that a `fts` directory.  Within that, add a json file for each index, with the file name being the index name.  For example, `/startup/default/fts/my_index.json`.  Note that names are case sensitive.

diff --git a/example/buckets.json b/example/buckets.json
@@ -2,10 +2,10 @@
   {
     "name": "sample",
     "ramQuotaMB": 100,
-    "bucketType": "membase",
+    "bucketType": "couchbase",
     "authType": "sasl",
     "saslPassword": "",
-    "evictionPolicy": "fullEviction",
+    "evictionPolicy": "valueOnly",
     "replicaNumber": 0,
     "flushEnabled": 1
   },
@@ -19,4 +19,4 @@
     "replicaNumber": 0,
     "flushEnabled": 1
   }
-]
+]
diff --git a/example/docker-compose.yml b/example/docker-compose.yml
@@ -4,7 +4,7 @@ services:
   couchbase:
     image: btburnett3/couchbasefakeit:enterprise-6.5.1
     environment:
-      - CB_SERVICES=kv,n1ql,index,fts,eventing
+      - CB_SERVICES=kv,n1ql,index,fts,eventing,cbas
     volumes:
       - "./:/startup"
     ports:

diff --git a/example/sample/analytics/dataset.json b/example/sample/analytics/dataset.json
@@ -0,0 +1,8 @@
+{
+  "statements": [
+    "CREATE DATAVERSE `sample` IF NOT EXISTS;",
+    "USE `sample`; CREATE DATASET IF NOT EXISTS users ON `sample` WHERE `type` = 'user';",
+    "USE `sample`; CONNECT LINK Local;"
+  ]
+}
+
diff --git a/example/sample/analytics/indexes.json b/example/sample/analytics/indexes.json
@@ -0,0 +1,5 @@
+{
+  "statements": [
+    "USE `sample`; CREATE INDEX `idx_users` IF NOT EXISTS ON `users` (id: string);"
+  ]
+}
diff --git a/scripts/configure-analytics.sh b/scripts/configure-analytics.sh
@@ -0,0 +1,27 @@
+#!/bin/bash
+
+bucketName=$1
+
+# Run Analytics Dataset scripts
+
+if [ -e "/startup/$bucketName/analytics/dataset.json" ]; then
+  echo "Creating datasets on $bucketName..."
+
+  while read statement
+  do
+    echo "Dataset Statement: $statement"
+    curl -Ss -u $CB_USERNAME:$CB_PASSWORD http://localhost:8095/analytics/service --data-urlencode "statement=$statement"
+  done < <(cat /startup/$bucketName/analytics/dataset.json | jq -r '.statements | .[]')
+fi
+
+# Run Analytics index scripts
+
+if [ -e "/startup/$bucketName/analytics/indexes.json" ]; then
+  echo "Building analytics indexes on $bucketName..."
+
+  while read statement
+  do
+    echo "Index Statement: $statement"
+    curl -Ss -u $CB_USERNAME:$CB_PASSWORD http://localhost:8095/analytics/service --data-urlencode "statement=$statement"
+  done < <(cat /startup/$bucketName/analytics/indexes.json | jq -r '.statements | .[]')
+fi
diff --git a/scripts/configure-node.sh b/scripts/configure-node.sh
@@ -19,6 +19,7 @@ if [ ! -e "/nodestatus/initialized" ] ; then
     $scriptPath/create-views.sh $bucketName
     $scriptPath/create-n1ql-indexes.sh $bucketName
     $scriptPath/create-fts-indexes.sh $bucketName
+    $scriptPath/configure-analytics.sh $bucketName
   done < <(cat /startup/buckets.json | jq -r '.[].name')
 
   $scriptPath/create-events.sh
@@ -32,4 +33,3 @@ fi
 
 # Wait for Couchbase Server shutdown
 fg 1
-
diff --git a/scripts/init-node.sh b/scripts/init-node.sh
@@ -14,7 +14,20 @@ sleep 5
 # Configure cluster, first request may need more time so retry
 for attempt in $(seq 60)
 do
-  curl -Ss http://127.0.0.1:8091/pools/default -d memoryQuota=$CB_DATARAM -d indexMemoryQuota=$CB_INDEXRAM -d ftsMemoryQuota=$CB_SEARCHRAM \
+  echo "Cluster name: $CB_CLUSTER_NAME"
+
+  # Change the name of the cluster if the user provided a cluster name
+  if [[ ! -z $CB_CLUSTER_NAME ]]; then
+    echo "Changing cluster name to $CB_CLUSTER_NAME..."
+    curl -Ss -X POST http://127.0.0.1:8091/pools/default -d clusterName=$CB_CLUSTER_NAME && echo
+  fi
+
+  curl -Ss -X POST http://127.0.0.1:8091/pools/default \
+      -d memoryQuota=$CB_DATARAM \
+      -d indexMemoryQuota=$CB_INDEXRAM \
+      -d ftsMemoryQuota=$CB_SEARCHRAM \
+      -d cbasMemoryQuota=$CB_ANALYTICSRAM \
+      -d eventingMemoryQuota=$CB_EVENTINGRAM \
     && echo \
     && break
 

diff --git a/scripts/is-the-bucket-ready.js b/scripts/is-the-bucket-ready.js
@@ -7,7 +7,7 @@ const couchbaseVersionSplit = 5;
 // This error code is expected so ignore it
 const errorCodeToIgnore = 22;
 
-var couchbase = require('couchbase')
+var couchbase = require('couchbase');
 var cluster = new couchbase.Cluster("http://127.0.0.1:8091");
 
 // If the couchbase version is 5 or greater then authenticate. For older versions this is unnecessary and will cause an error
@@ -22,7 +22,7 @@ var bucket = cluster.openBucket(bucketName, err => {
   }
 });
 
-var key = "ping"
+var key = "ping";
 bucket.upsert(key, "ping", err => {
   if (err) {
     if (err.code !== errorCodeToIgnore) {
@@ -32,8 +32,8 @@ bucket.upsert(key, "ping", err => {
   }
 
   // Don't keep the ping document in the bucket
-  bucket.remove(key, err => {
-    if (err) {
+  bucket.remove(key, error => {
+    if (error) {
       console.log(`${key} document was not removed`);
       process.exit(1);
     }

diff --git a/scripts/wait-for-services.sh b/scripts/wait-for-services.sh
@@ -20,19 +20,31 @@ if [[ $CB_SERVICES == *"fts"* ]]; then
   # Wait for the FTS service to be up and running
   for attempt in $(seq 10)
   do
-    curl -s http://127.0.0.1:8094/api/ping > /dev/null \
+    curl -s -u $CB_USERNAME:$CB_PASSWORD http://127.0.0.1:8094/api/index > /dev/null \
       && break
 
     echo "Waiting for FTS service..."
     sleep 1
   done
 fi
 
+if [[ $CB_SERVICES == *"cbas"* ]]; then
+  # Wait for the analytics service to be up and running
+  for attempt in $(seq 10)
+  do
+    curl -s -u $CB_USERNAME:$CB_PASSWORD http://localhost:8095/analytics/config/service > /dev/null \
+      && break
+
+    echo "Waiting for analytics service..."
+    sleep 1
+  done
+fi
+
 if [[ $CB_SERVICES == *"eventing"* ]]; then
   # Wait for the eventing service to be up and running
   for attempt in $(seq 10)
   do
-    curl -s http://127.0.0.1:8096/api/v1/functions > /dev/null \
+    curl -s -u $CB_USERNAME:$CB_PASSWORD http://127.0.0.1:8096/api/v1/functions > /dev/null \
       && break
 
     echo "Waiting for eventing service..."

diff --git a/startup/buckets.json b/startup/buckets.json
@@ -2,11 +2,11 @@
   {
     "name": "default",
     "ramQuotaMB": 100,
-    "bucketType": "membase",
+    "bucketType": "couchbase",
     "authType": "sasl",
     "saslPassword": "",
-    "evictionPolicy": "fullEviction",
+    "evictionPolicy": "valueOnly",
     "replicaNumber": 0,
     "flushEnabled": 1
   }
-]
+]