Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Solr TW review #1153

Closed
wants to merge 9 commits into from
Closed

Solr TW review #1153

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
89e8b18
Update README.md
GeoFrau Jan 8, 2021
24d8440
Update schema.yaml
GeoFrau Jan 8, 2021
c7db650
Update application.yaml
GeoFrau Jan 12, 2021
5b0c080
Merge branch 'master' into Solr-TW-review
GeoFrau Jan 12, 2021
8a0816a
Merge branch 'master' into Solr-TW-review
farajn9 Jan 21, 2021
80542f2
Merge branch 'master' into Solr-TW-review
huyhg Jan 29, 2021
ff501f2
Merge branch 'master' into Solr-TW-review
ovk6 Mar 8, 2021
e7cf2f8
Merge branch 'master' into Solr-TW-review
GeoFrau Sep 12, 2022
eb70e5a
Merge branch 'master' into Solr-TW-review
armandomiani Aug 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 99 additions & 51 deletions k8s/solr/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
# Overview

Apache Solr is an open-source enterprise-search platform, written in Java.
Apache Solr is an open-source enterprise search platform, written in Java,
from the Apache Lucene project.

For more information on Apache Solr, see the Apache Solr [official website](https://solr.apache.org/).
For more information on Apache Solr, see the Apache Solr
[official website](https://lucene.apache.org/solr/).

## About Google Click to Deploy

Expand All @@ -12,41 +14,49 @@ Popular open stacks on Kubernetes, packaged by Google.

![Architecture diagram](resources/apache-solr-k8s-app-architecture.png)

This solution is focused on SolrCloud mode which uses ZooKeeper as a external repository. By default 3 replicas of SolrCloud nodes and 3 replicas of ZooKeeper nodes are deployed. For more information about SolrCloud visit [official documentation](https://lucene.apache.org/solr/guide/8_6/solrcloud.html).
This solution is focused on SolrCloud mode, which uses ZooKeeper as a external
repository. By default, 3 replicas of SolrCloud nodes and 3 replicas of ZooKeeper
nodes are deployed. For more information about SolrCloud, visit the
[official documentation](https://lucene.apache.org/solr/guide/8_6/solrcloud.html).

[Basic authentication plugin](https://solr.apache.org/guide/solr/latest/deployment-guide/basic-authentication-plugin.html) is enabled for SolrCloud by Job resource during post deployment.
After you deploy this solution, the Job resource enables the
[basic authentication plugin](https://lucene.apache.org/solr/guide/8_6/basic-authentication-plugin.html)
for SolrCloud.

[ZooKeeper Access Control](https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html) also applied to protect SolrCloud contents in ZooKeeper.
The solution uses
[ZooKeeper Access Control](https://lucene.apache.org/solr/guide/8_6/zookeeper-access-control.html)
to protect its SolrCloud content in ZooKeeper.

# Installation
# Installing the solution

## Installing with Google Cloud Marketplace

Get up and running with a few clicks! To install this SolrCloud app to a Google
Kubernetes Engine (GKE) cluster by using Google Cloud Marketplace, follow the
Get up and running with a few clicks! To install this SolrCloud app to a
Google Kubernetes Engine (GKE) cluster by using Google Cloud Marketplace,
follow the
[on-screen instructions](https://console.cloud.google.com/marketplace/details/google/solr).

## Command-line instructions
## Installing with the command line

You can use [Google Cloud Shell](https://cloud.google.com/shell/) or a local
workstation to follow the steps below.

[![Open in Cloud Shell](http://gstatic.com/cloudssh/images/open-btn.svg)](https://console.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/GoogleCloudPlatform/click-to-deploy&cloudshell_open_in_editor=README.md&cloudshell_working_dir=k8s/solr)

### Prerequisites
### Before you begin

#### Setting up command-line tools

You'll need the following tools in your development environment. If you are
using Cloud Shell, then `gcloud`, `kubectl`, Docker, and Git are installed in your
environment by default.
using Cloud Shell, then `gcloud`, `kubectl`, Docker, and Git are installed in
your environment by default.

* [gcloud](https://cloud.google.com/sdk/gcloud/)
* [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)
* [docker](https://docs.docker.com/install/)
* [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
* [openssl](https://www.openssl.org/)
* [helm](https://helm.sh/)
* [Docker](https://docs.docker.com/install/)
* [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
* [OpenSSL](https://www.openssl.org/)
* [Helm](https://helm.sh/)

Configure `gcloud` as a Docker credential helper:

Expand Down Expand Up @@ -92,7 +102,7 @@ command:
kubectl apply -f "https://raw.githubusercontent.com/GoogleCloudPlatform/marketplace-k8s-app-tools/master/crd/app-crd.yaml"
```

You need to run this command once.
You must run this command once.

The
[Kubernetes SIG-apps](https://github.com/kubernetes/community/tree/master/sig-apps)
Expand All @@ -116,13 +126,14 @@ use the `default` namespace.
export APP_INSTANCE_NAME=solr-1
export NAMESPACE=default
```
(Optional) Enable Stackdriver Metrics Exporter:
(Optional) Enable Cloud Monitoring Metrics Exporter:

> **NOTE:** Your GCP project must have Stackdriver enabled. If you are using a
> non-GCP cluster, you cannot export metrics to Stackdriver.
> **NOTE:** Your Google Cloud project must have Google Cloud's operations
> suite enabled. If you are using a non-Google Cloud cluster, you cannot export
> metrics to Google Cloud's operations suite.

By default, the application does not export metrics to Stackdriver. To enable
this option, change the value to `true`.
By default, the app does not export metrics to Google Cloud's operations suite.
To enable this option, change the value to `true`.

```shell
export METRICS_EXPORTER_ENABLED=false
Expand Down Expand Up @@ -160,10 +171,11 @@ export ZK_ADMIN_USER="admin-user"
export ZK_ADMIN_PASSWORD="$(generate_pwd)"
```

Set the storage class for the persistent volume of SolrCloud nodes and ZooKeeper nodes:
Set the storage class for the persistent volume of SolrCloud nodes and
ZooKeeper nodes:

* Set the StorageClass name. You can select your existing StorageClass name for
the persistent disk of SolrCloud application storage.
the persistent disk of SolrCloud app storage.
* Set the persistent disk's size for SolrCloud storage. The default disk size is
`10Gi`.
* Set the persistent disk's size for ZooKeeper storage. The default disk size
Expand All @@ -174,10 +186,11 @@ export STORAGE_CLASS="standard" # provide your StorageClass name if not "standar
export PERSISTENT_SOLR_SIZE="10Gi"
export PERSISTENT_ZK_SIZE="5Gi"
```

#### Creating a namespace in your Kubernetes cluster

If you use a different namespace than `default`, or if the namespace does
not exist yet, create a new namespace:
If you use a different namespace than `default`, or if the namespace does not
exist yet, create a new namespace:

```shell
kubectl create namespace "${NAMESPACE}"
Expand Down Expand Up @@ -215,9 +228,10 @@ kubectl apply -f "${APP_INSTANCE_NAME}_manifest.yaml" --namespace "${NAMESPACE}"

### Accessing the SolrCloud UI web console

The deployed service of SolrCloud is of type ClusterIP, so you can reach the SolrCloud Studio UI within a Kubernetes cluster by port-forwarding.
The deployed service of SolrCloud is of type ClusterIP, so you can reach the
SolrCloud Studio UI within a Kubernetes cluster by port-forwarding.

To accomplish this, run the following commands:
To do this, run the following commands:

```shell
# Get "solr" user credentials of SolrCloud UI
Expand All @@ -233,11 +247,13 @@ echo "password: ${SOLR_PASSWORD}"
kubectl port-forward svc/${APP_INSTANCE_NAME}-solr-svc --namespace ${NAMESPACE} 8983
```

After running the commands, visit [http://localhost:8983](http://localhost:8983)
from a web browser. Login with `solr` credentials.
After you've run the commands, visit
[http://localhost:8983](http://localhost:8983) from a web browser. Login with
your `solr` credentials.

### Interacting with SolrCloud via `curl`
> Note: Port-forwarding should be kept running

> Note: You should keep port-forwarding running while you follow these steps.

```shell
# Check Cluster status
Expand All @@ -251,8 +267,11 @@ curl --user ${SOLR_USER}:${SOLR_PASSWORD} "http://localhost:8983/solr/admin/coll

### Interacting with SolrCloud via `solr` control script

For example if you have collection created you can check healtcheck by help of Solr control script.
> Visit [Solr Control Script Reference](https://solr.apache.org/guide/solr/latest/deployment-guide/solr-control-script-reference.html) for more command options
For example, if you have created a collection, you can check its health by
using a Solr control script.

> For more command options, visit the
> [Solr Control Script Reference](https://lucene.apache.org/solr/guide/8_6/solr-control-script-reference.html).

```shell
# Use your collection name instead of "my_collection"
Expand All @@ -262,11 +281,17 @@ kubectl -n ${NAMESPACE} exec -it ${APP_INSTANCE_NAME}-solr-0 -- bash -c "solr he
```

## Authentication and Security
During post deployment of this SolrCloud solution, Solr basic authentication is enabled by Job resource. This action creates `security.json` and uploads it to ZooKeeper.

To protect this configuration files in ZooKeeper, ZooKeeper ACL with "admin-user" and "readonly-user" is configured and used by Solr instances.
After you deploy this SolrCloud solution, the Job resource enables Solr basic
authentication. This creates `security.json` and uploads it to ZooKeeper.

To protect this configuration files in ZooKeeper, ZooKeeper Acceess Control
Lists (ACLs) with "admin-user" and "readonly-user" are configured and
used by Solr instances.

You pass these Java parameters to external clients who want to connect to
the secure SolrCloud. To check these parameters, run the following commands:

Those Java parameters should be passed to external clients which want to connect to secure SolrCloud. You can check this parameters by running below commands:
```shell
# Java option for basic auth
kubectl -n ${NAMESPACE} exec -it ${APP_INSTANCE_NAME}-solr-0 -- bash -c "echo SOLR_AUTH_TYPE=\$SOLR_AUTH_TYPE SOLR_AUTHENTICATION_OPTS=\$SOLR_AUTHENTICATION_OPTS"
Expand All @@ -276,16 +301,26 @@ kubectl -n ${NAMESPACE} exec -it ${APP_INSTANCE_NAME}-solr-0 -- bash -c "echo SO
```

# Monitoring
Solr image includes a Prometheus exporter to collect metrics and other data. In this solution seperate deployment and service used to access metrics. Exporter Deployment uses same Solr image, connects to main SolrCloud and ZooKeeper service, authenticates and exports metrics at 9983 port for `/metrics` endpoint.

> Visit [Monitoring Solr with Prometheus and Grafana documentation](https://solr.apache.org/guide/solr/latest/deployment-guide/monitoring-with-prometheus-and-grafana.html) for more information.
This Solr image includes a Prometheus exporter to collect metrics and other
data. In this solution, a seperate deployment and service are used to access
metrics. The exporter deployment uses same Solr image, connects to the main
SolrCloud and ZooKeeper service, and authenticates and exports metrics at the
`/metrics` endpoint port 9983.

> For more information, visit the
> [Monitoring Solr with Prometheus and Grafana documentation](https://lucene.apache.org/solr/guide/8_6/monitoring-solr-with-prometheus-and-grafana.html).

To check metrics from the Solr exporter service, forward the exporter service's
port to your local machine:

To check metrics from Solr exporter service forward exporter service port to local machine
```shell
# Forward Solr exporter service port to local workspace
kubectl port-forward svc/${APP_INSTANCE_NAME}-solr-exporter --namespace ${NAMESPACE} 9983
```
Visit http://localhost:9983/metrics to see metrics from Solr.

After you forward the port, to see metrics from Solr, visit
http://localhost:9983/metrics.

# Scaling

Expand All @@ -306,11 +341,16 @@ When this option is used to scale down a cluster, it reduces the number of
replicas without disconnecting nodes from the cluster. Scaling down also does
not affect the `persistentvolumeclaims` of your StatefulSet.

> For information on enabling autoscaling for collections, visit the
> [SolrCloud Autoscaling documentation](https://lucene.apache.org/solr/guide/8_6/solrcloud-autoscaling-auto-add-replicas.html).

# Upgrading the app

Before you upgrade the app, we recommend that you back up all of your
SolrCloud collections.
> Visit [Solr Upgrade Notes](https://solr.apache.org/guide/solr/latest/upgrade-notes/solr-upgrade-notes.html) to check limitations for upgrading between versions

> To check limitations on upgrading between versions, visit the
> [Solr Upgrade Notes](https://lucene.apache.org/solr/guide/8_6/solr-upgrade-notes.html).

The Solr StatefulSet is configured to roll out updates automatically. To
start the update, patch the StatefulSet with a new image reference, where
Expand All @@ -329,14 +369,21 @@ new image, run the following command:
kubectl get pods --selector app.kubernetes.io/name=${APP_INSTANCE_NAME} \
--namespace ${NAMESPACE}
```

# Solr Standalone mode
It is also possible to deploy this Solr application as a standalone mode for testing purposes.
> Warning: This should not be used for production!

With Standalone mode enabled, instead of external ZooKeeper ensemble embedded ZooKeeper is used and Solr is deployed with single replica only.
Also no any authentication will be applied to the solution to make it easier for testing.
It is also possible to deploy this Solr app in a standalone mode, for
testing purposes.

> Warning: This should not be used for production.

With Standalone mode enabled, the app uses embedded ZooKeeper instead of an
external ZooKeeper ensemble, and deploys Solr with a single replica only.
The solution also does not apply any authentication features, to enable more
convenient testing.

To deploy Solr in standalone mode, run the following commands:

To deploy Solr as a standalone mode, run following commands:
```shell
# Export necessary variables
APP_INSTANCE_NAME=solr-standalone
Expand Down Expand Up @@ -390,8 +437,8 @@ export NAMESPACE=default
### Deleting the resources

> Note: We recommend using the version of `kubectl` that is the same as the
> version for your cluster. Using the same version for `kubectl` and the cluster
> helps to avoid unforeseen issues.
> version for your cluster. Using the same version for `kubectl` and the
> cluster helps to avoid unforeseen issues.

#### Deleting the deployment with the generated manifest file

Expand All @@ -412,9 +459,10 @@ kubectl delete application,statefulset,secret,service,deployment,pdb,cm,jobs \
--selector app.kubernetes.io/name=${APP_INSTANCE_NAME}
```

Deleting the `Application` resource deletes all of your deployment's resources,
except for `PersistentVolumeClaim`. To remove the `PersistentVolumeClaim`s
with their attached persistent disks, run the following `kubectl` command:
When you delete the `Application` resource, you delete all of your deployment's
resources, except for `PersistentVolumeClaim`. To remove the
`PersistentVolumeClaim`s with their attached persistent disks, run the following
`kubectl` command:

```shell
kubectl delete persistentvolumeclaims \
Expand Down
12 changes: 6 additions & 6 deletions k8s/solr/chart/solr/templates/application.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ spec:
type: {{ template "solrMode" . }}
version: {{ .Values.solr.image.tag | quote }}
description: |-
Apache Solr is an open-source enterprise-search platform, written in Java, from the Apache Lucene project.
Apache Solr is an open-source enterprise search platform, written in Java, from the Apache Lucene project.

# Support
Google does not offer support for this solution. However, community support is available on [StackOverflow](https://stackoverflow.com/questions/tagged/solr).
Expand Down Expand Up @@ -45,11 +45,11 @@ spec:

Make sure that all components are in `ready` status.

Deployed service of {{ template "solrMode" . }} is ClusterIP type, so you can reach to {{ template "solrMode" . }} Admin UI within a kubernetes cluster.
Deployed service of {{ template "solrMode" . }} is of type ClusterIP, so you can reach to {{ template "solrMode" . }} Admin UI within a Kubernetes cluster.

{{- if not .Values.solr.standalone }}
SolrCloud is configured with [Basic authentication plugin](https://lucene.apache.org/solr/guide/8_6/basic-authentication-plugin.html) beside [ZooKeeper ACL](https://lucene.apache.org/solr/guide/8_6/zookeeper-access-control.html).
You can check credentials from application page in console or you can print it by running below commands:
SolrCloud is configured with both a [basic authentication plugin](https://lucene.apache.org/solr/guide/8_6/basic-authentication-plugin.html) and [ZooKeeper access control lists (ACL)](https://lucene.apache.org/solr/guide/8_6/zookeeper-access-control.html).
You can check the credentials from the app's page in the console, or you print them by running the following commands:

```shell
# Get and print SolrCloud user credentials
Expand All @@ -64,7 +64,7 @@ spec:
```
{{- else }}

Solr deployed as a standalone mode without any authentication enabled and for development and testing purposes.
Solr deployed in a standalone mode, without any authentication enabled, for development and testing purposes.

{{- end }}

Expand All @@ -75,7 +75,7 @@ spec:
--namespace {{ .Release.Namespace }} 8983
```

Then visit [http://localhost:8983/solr](http://localhost:8983/solr) on your web browser and login with `solr` user credentials if necessary.
Visit [http://localhost:8983/solr](http://localhost:8983/solr) on your web browser and log in with `solr` user credentials, if necessary.
info:
- name: {{ template "solrMode" . }} service
type: Reference
Expand Down
8 changes: 4 additions & 4 deletions k8s/solr/schema.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,15 +54,15 @@ properties:
persistence.storageClass:
type: string
title: StorageClass
description: You can choose an existing StorageClass or create new one.
description: You can choose an existing StorageClass or create a new one.
x-google-marketplace:
type: STORAGE_CLASS
storageClass:
type: SSD
solr.standalone:
type: boolean
title: Solr Standalone mode
description: Solr Standalone mode is mainly for development and testing purposes. If this option is enabled external ZooKeeper ensemble will not be deployed and authentication will be disabled for the Solr services.
description: Solr Standalone mode is mainly for development and testing purposes. If this option is enabled, external ZooKeeper Ensemble will not be deployed and authentication will be disabled for the Solr services.
default: false

persistence.solr.storage.size:
Expand Down Expand Up @@ -104,8 +104,8 @@ properties:

metrics.exporter.enabled:
type: boolean
title: Enable Stackdriver Metrics Exporter
description: Your GCP project must have Stackdriver enabled. Non-GCP clusters cannot export metrics to Stackdriver.
title: Enable Cloud Monitoring Metrics Exporter
description: Your Google Cloud project must have Google Cloud's operations suite enabled. Non-Google Cloud clusters cannot export metrics to Cloud Monitoring.
default: false

required:
Expand Down
Loading