asadiqbal08/Migrating from ElasticSearch to OpenSearch (#5166)

* Moving from ElasticSearch to OpenSearch and code refactoring

* Update indexing_api_test.py

Removed the unnecessary comment

.env.example

@@ -2,7 +2,7 @@ STATUS_TOKEN=
 EDXORG_BASE_URL=
 EDXORG_CLIENT_ID=
 EDXORG_CLIENT_SECRET=
-ELASTICSEARCH_INDEX=
+OPENSEARCH_INDEX=
 CYBERSOURCE_ACCESS_KEY=
 CYBERSOURCE_SECURITY_KEY=
 CYBERSOURCE_TRANSACTION_KEY=
@@ -20,5 +20,5 @@ MAILGUN_KEY=
 MAILGUN_RECIPIENT_OVERRIDE=
 MAILGUN_FROM_EMAIL=
 MAILGUN_URL=
-ELASTICSEARCH_INDEX=
-ELASTICSEARCH_INDEXING_CHUNK_SIZE=100
+OPENSEARCH_INDEX=
+OPENSEARCH_INDEXING_CHUNK_SIZE=100

.github/workflows/ci.yml

@@ -27,8 +27,8 @@ jobs:
         ports:
           - 6379:6379
 
-      elastic:
-        image: docker.elastic.co/elasticsearch/elasticsearch:6.7.1
+      opensearch:
+        image: opensearchproject/opensearch:1.2.4
         env:
           network.host: "0.0.0.0"
           http.cors.enabled: "true"
@@ -38,6 +38,9 @@ jobs:
         ports:
           - 9200:9200
 
+    strategy:
+      matrix:
+        opensearch-version: [1.2]
     steps:
       - uses: actions/checkout@v2
 
@@ -63,18 +66,18 @@ jobs:
       - name: Lint
         run: pylint ./**/*.py
 
-      # You must also add the Configure sysctl limits step, otherwise Elasticsearch will not be able to boot.
+      # You must also add the Configure sysctl limits step, otherwise Opensearch will not be able to boot.
       - name: Configure sysctl limits
         run: |
           sudo swapoff -a
           sudo sysctl -w vm.swappiness=1
           sudo sysctl -w fs.file-max=262144
           sudo sysctl -w vm.max_map_count=262144
 
-      - name: Runs Elasticsearch
-        uses: elastic/elastic-github-actions/elasticsearch@master
+      - name: Runs Opensearch
+        uses: ankane/setup-opensearch@v1
         with:
-          stack-version: 6.7.1
+          opensearch-version: ${{ matrix.opensearch-version }}
 
       - name: Tests
         run: |
@@ -88,7 +91,7 @@ jobs:
           DATABASE_URL: postgres://postgres:postgres@localhost:5432/postgres
           MICROMASTERS_SECURE_SSL_REDIRECT: 'False'
           MICROMASTERS_DB_DISABLE_SSL: 'True'
-          ELASTICSEARCH_URL: localhost:9200
+          OPENSEARCH_URL: localhost:9200
           CELERY_TASK_ALWAYS_EAGER: 'True'
           BROKER_URL: redis://localhost:6379/4
           CELERY_RESULT_BACKEND: redis://localhost:6379/4

README.md

@@ -8,7 +8,7 @@ Portal for learners and course teams to access MITx MicroMasters programs.
 MicroMasters mostly follows the same [initial setup steps outlined in the common ODL web app guide](https://github.com/mitodl/handbook/blob/master/common-web-app-guide.md).
 Run through those steps in order with the following changes:
 
-1. After [creating the database models](https://github.com/mitodl/handbook/blob/master/common-web-app-guide.md#3-create-database-tables-from-the-django-models), you should create the Elasticsearch indices via management command:
+1. After [creating the database models](https://github.com/mitodl/handbook/blob/master/common-web-app-guide.md#3-create-database-tables-from-the-django-models), you should create the Opensearch indices via management command:
 
    ```
    docker-compose run web ./manage.py recreate_index
@@ -213,20 +213,20 @@ To validate prices and financial aid discounts for all programs run:
 
 # Connecting to external services
 
-#### Elasticsearch
+#### Opensearch
 
 If you want to connect to an ES cluster aside from the one created by Docker, you'll need to do the following:
 
 1. Add these variables to your `.env` file (without parentheses):
 
-        ELASTICSEARCH_INDEX=(your_index_name)
-        ELASTICSEARCH_URL=https://(your_elastic_search_url)
-        ELASTICSEARCH_HTTP_AUTH=(your_cluster_name):(key)
+        OPENSEARCH_INDEX=(your_index_name)
+        OPENSEARCH_URL=https://(your_open_search_url)
+        OPENSEARCH_HTTP_AUTH=(your_cluster_name):(key)
 
 2. If any of the above variables are set in the `web` configuration in `docker-compose.yml`, those
  will override the values you have in `.env`. Delete them.
-3. Restart the `db` and `elastic` docker-compose services if they're running:
- `docker-compose restart db elastic`
+3. Restart the `db` and `opensearch-node1` docker-compose services if they're running:
+ `docker-compose restart db opensearch-node1`
 
 You should now be able to connect to the external ES cluster. You
 can run `docker-compose run web ./manage.py recreate_index` to test

RELEASE.rst

@@ -1780,7 +1780,7 @@ Version 0.79.0 (Released October 31, 2017)
 - Doc about how to freeze final grades (#3658)
 - Use yarn install --frozen-lockfile to error if upgrade needed (#3653)
 - Refactor docker-compose.yml files (#3644)
-- Use HEROKU_APP_NAME as ELASTICSEARCH_INDEX value for PR builds (#3640)
+- Use HEROKU_APP_NAME as OPENSEARCH_INDEX value for PR builds (#3640)
 
 Version 0.78.1 (Released October 20, 2017)
 --------------

app.json

@@ -72,21 +72,25 @@
       "description": "The OAuth client secret configured in the edX instance.",
       "required": true
     },
-    "ELASTICSEARCH_HTTP_AUTH": {
-      "description": "Basic auth settings for connecting to Elasticsearch"
+    "OPENSEARCH_HTTP_AUTH": {
+      "description": "Basic auth settings for connecting to Opensearch",
+      "required": false
     },
-    "ELASTICSEARCH_SHARD_COUNT": {
-      "description": "Configurable shard cound for Elasticsearch"
+    "OPENSEARCH_SHARD_COUNT": {
+      "description": "Configurable shard cound for Opensearch",
+      "required": false
     },
-    "ELASTICSEARCH_INDEXING_CHUNK_SIZE": {
-      "description": "Chunk size to use for Elasticsearch indexing tasks",
+    "OPENSEARCH_INDEXING_CHUNK_SIZE": {
+      "description": "Chunk size to use for Opensearch indexing tasks",
       "required": false
     },
-    "ELASTICSEARCH_INDEX": {
-      "description": "Index to use on Elasticsearch"
+    "OPENSEARCH_INDEX": {
+      "description": "Index to use on Opensearch",
+      "required": false
     },
-    "ELASTICSEARCH_URL": {
-      "description": "URL for connecting to Elasticsearch cluster"
+    "OPENSEARCH_URL": {
+      "description": "URL for connecting to Opensearch cluster",
+      "required": false
     },
     "FEATURE_OPEN_DISCUSSIONS_USER_SYNC": {
       "description": "Enables creation and syncing of open-discussions user data",

certificates/views_test.py

@@ -23,7 +23,7 @@
 )
 
 pytestmark = [
-    pytest.mark.usefixtures('mocked_elasticsearch'),
+    pytest.mark.usefixtures('mocked_opensearch'),
     pytest.mark.django_db,
 ]
 

conftest.py

@@ -71,11 +71,12 @@ def settings_defaults(settings):
     settings.FEATURES['OPEN_DISCUSSIONS_USER_SYNC'] = False
 
 
-@pytest.fixture(scope='module')
-def mocked_elasticsearch_module_patcher():
+@pytest.fixture(scope='function')
+def mocked_opensearch_module_patcher(settings):
     """
-    Fixture that patches all indexing API functions that communicate directly with ElasticSearch
+    Fixture that patches all indexing API functions that communicate directly with OpenSearch
     """
+    settings.DEBUG=True
     patchers = []
     patcher_mocks = []
     for name, val in tasks.__dict__.items():
@@ -96,13 +97,13 @@ def mocked_elasticsearch_module_patcher():
 
 
 @pytest.fixture()
-def mocked_elasticsearch(mocked_elasticsearch_module_patcher):
+def mocked_opensearch(mocked_opensearch_module_patcher):
     """
-    Fixture that resets all of the patched ElasticSearch API functions
+    Fixture that resets all of the patched OpenSearch API functions
     """
-    for mock in mocked_elasticsearch_module_patcher.patcher_mocks:
+    for mock in mocked_opensearch_module_patcher.patcher_mocks:
         mock.reset_mock()
-    return mocked_elasticsearch_module_patcher
+    return mocked_opensearch_module_patcher
 
 
 @pytest.fixture()

courses/catalog_serializers_test.py

@@ -12,7 +12,7 @@
 
 
 @pytest.mark.django_db
-@pytest.mark.usefixtures("mocked_elasticsearch")
+@pytest.mark.usefixtures("mocked_opensearch")
 @pytest.mark.parametrize("has_page", [True, False])
 @pytest.mark.parametrize("has_thumbnail", [True, False])
 def test_catalog_program_serializer(has_page, has_thumbnail):

discussions/api_test.py

@@ -2,7 +2,7 @@
 # pylint: disable=redefined-outer-name
 from django.core.exceptions import ImproperlyConfigured
 from django.db.models.signals import post_save
-from elasticsearch_dsl import Search
+from opensearch_dsl import Search
 from factory.django import mute_signals
 from open_discussions_api.constants import ROLE_STAFF
 import pytest
@@ -43,7 +43,7 @@
 )
 
 pytestmark = [
-    pytest.mark.usefixtures('mocked_elasticsearch'),
+    pytest.mark.usefixtures('mocked_opensearch'),
     pytest.mark.usefixtures('mocked_on_commit'),
     pytest.mark.django_db,
 ]

discussions/signals_test.py

@@ -9,7 +9,7 @@
 from roles.factories import RoleFactory
 
 pytestmark = [
-    pytest.mark.usefixtures('mocked_elasticsearch'),
+    pytest.mark.usefixtures('mocked_opensearch'),
     pytest.mark.usefixtures('mocked_on_commit'),
     pytest.mark.django_db,
 ]

discussions/tasks_test.py

@@ -15,7 +15,7 @@
 )
 
 pytestmark = [
-    pytest.mark.usefixtures('mocked_elasticsearch'),
+    pytest.mark.usefixtures('mocked_opensearch'),
     pytest.mark.usefixtures('mocked_on_commit'),
     pytest.mark.django_db,
 ]

discussions/views_test.py

@@ -15,7 +15,7 @@
 from roles.models import Staff
 
 pytestmark = [
-    pytest.mark.usefixtures('mocked_elasticsearch'),
+    pytest.mark.usefixtures('mocked_opensearch'),
     pytest.mark.usefixtures('mocked_on_commit'),
     pytest.mark.django_db,
 ]

docker-compose.selenium.yml

@@ -11,15 +11,15 @@ services:
       DATABASE_URL: postgres://postgres:postgres@db:5432/postgres
       MICROMASTERS_SECURE_SSL_REDIRECT: 'False'
       MICROMASTERS_DB_DISABLE_SSL: 'True'
-      ELASTICSEARCH_URL: elastic:9200
+      OPENSEARCH_URL: opensearch-node1:9200
       CELERY_ALWAYS_EAGER: 'False'
       BROKER_URL: redis://redis:6379/4
       CELERY_RESULT_BACKEND: redis://redis:6379/4
       DOCKER_HOST: ${DOCKER_HOST:-missing}
       WEBPACK_DEV_SERVER_HOST: ${WEBPACK_DEV_SERVER_HOST:-localhost}
-      ELASTICSEARCH_INDEX: 'testindex'
+      OPENSEARCH_INDEX: 'testindex'
       DEBUG: 'False'
-      ELASTICSEARCH_DEFAULT_PAGE_SIZE: '5'
+      OPENSEARCH_DEFAULT_PAGE_SIZE: '5'
       MITXONLINE_BASE_URL: "https://fake-mitxonline.example.com/"
       MITXONLINE_URL: "https://fake-mitxonline.example.com/"
       # To silence ImproperlyConfigured when running tests
@@ -28,7 +28,7 @@ services:
       OPEN_DISCUSSIONS_SITE_KEY: fake_site_key
     links:
       - db
-      - elastic
+      - opensearch-node1
       - redis
       - hub
       - chrome

docker-compose.yml

@@ -8,7 +8,7 @@ x-environment:
   DATABASE_URL: postgres://postgres:postgres@db:5432/postgres
   MICROMASTERS_SECURE_SSL_REDIRECT: 'False'
   MICROMASTERS_DB_DISABLE_SSL: 'True'
-  ELASTICSEARCH_URL: elastic:9200
+  OPENSEARCH_URL: opensearch-node1:9200
   CELERY_ALWAYS_EAGER: 'False'
   BROKER_URL: redis://redis:6379/4
   CELERY_RESULT_BACKEND: redis://redis:6379/4
@@ -31,12 +31,29 @@ services:
     ports:
       - "6379"
 
-  elastic:
-    image: elasticsearch:6.7.1
-    command: elasticsearch -E network.host=0.0.0.0 -E http.cors.enabled=true -E http.cors.allow-origin=* -E rest.action.multi.allow_explicit_index=false
-    user: elasticsearch
+  opensearch-node1:
+    image: opensearchproject/opensearch:1.2.4
+    container_name: opensearch-node1
+    environment:
+      - cluster.name=opensearch-cluster
+      - node.name=opensearch-node1
+      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
+      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
+      - "DISABLE_INSTALL_DEMO_CONFIG=true" # disables execution of install_demo_configuration.sh bundled with security plugin, which installs demo certificates and security configurations to OpenSearch
+      - "DISABLE_SECURITY_PLUGIN=true" # disables security plugin entirely in OpenSearch by setting plugins.security.disabled: true in opensearch.yml
+      - "discovery.type=single-node" # disables bootstrap checks that are enabled when network.host is set to a non-loopback address
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nofile:
+        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
+        hard: 65536
+    volumes:
+      - opensearch-data1:/usr/share/opensearch/data
     ports:
-      - "9100:9200"
+      - 9100:9200
+      - 9600:9600 # required for Performance Analyzer
 
   nginx:
     image: nginx:1.9.5
@@ -65,7 +82,7 @@ services:
       - "8077:8077"
     links:
       - db
-      - elastic
+      - opensearch-node1
       - redis
       - sftp
     extra_hosts:
@@ -98,7 +115,7 @@ services:
       celery -A micromasters.celery:app worker -Q search,exams,dashboard,default -B -l ${MICROMASTERS_LOG_LEVEL:-INFO}'
     links:
       - db
-      - elastic
+      - opensearch-node1
       - redis
       - sftp
 
@@ -107,3 +124,6 @@ services:
     ports:
         - "2022:22"
     command: odl:123:1001:1001:results,results/topvue
+
+volumes:
+  opensearch-data1:

financialaid/views_pytest_test.py

@@ -23,7 +23,7 @@
 
 
 pytestmark = [
-    pytest.mark.usefixtures('mocked_elasticsearch'),
+    pytest.mark.usefixtures('mocked_opensearch'),
     pytest.mark.django_db,
 ]
 

grades/tasks_pytest_test.py

@@ -17,7 +17,7 @@
 from micromasters.utils import now_in_utc
 
 pytestmark = [
-    pytest.mark.usefixtures('mocked_elasticsearch'),
+    pytest.mark.usefixtures('mocked_opensearch'),
     pytest.mark.django_db,
 ]
 

mail/api_test.py

@@ -10,7 +10,7 @@
 from django.core.exceptions import ImproperlyConfigured
 from django.db.models.signals import post_save
 from django.test import override_settings
-from elasticsearch_dsl import Search
+from opensearch_dsl import Search
 from factory.django import mute_signals
 from requests import Response
 from requests.exceptions import HTTPError