From 7618ec9f03e42a477e64097e414911184e0120a6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9gis=20Behmo?= Date: Thu, 17 Oct 2024 08:35:13 +0200 Subject: [PATCH 1/6] docs: *.local.edly.io -> *.local.openedx.io The default URL to run a local platform switched from local.edly.io to local.openedx.io. This changes makes it clearer for everyone that Tutor is to run Open edX. See: https://github.com/overhangio/tutor/issues/1120 --- README.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.rst b/README.rst index a904db5..967111d 100644 --- a/README.rst +++ b/README.rst @@ -38,7 +38,7 @@ Operations Creating a user ~~~~~~~~~~~~~~~ -The discovery user interface will be available at http://discovery.local.edly.io for a local test instance, +The discovery user interface will be available at http://discovery.local.openedx.io for a local test instance, and at ``DISCOVERY_HOST`` (by default: http(s)://discovery.) in production. To run commands from the UI, a user must be created: @@ -46,7 +46,7 @@ commands from the UI, a user must be created: tutor local run discovery ./manage.py createsuperuser -Then, you must log in with this user at http://discovery.local.edly.io/admin. +Then, you must log in with this user at http://discovery.local.openedx.io/admin. Alternatively, you can log in with oauth2 using a pre-existing user created on the LMS/CMS by accessing http(s)://discovery./login. To do so, the proper domain names must exist and point to @@ -108,13 +108,13 @@ extra argument to the command. i.e. ``--domain``. While running tutor in product .. code-block:: bash - tutor local run lms ./manage.py lms cache_programs --domain="local.edly.io" + tutor local run lms ./manage.py lms cache_programs --domain="local.openedx.io" While running tutor in development mode: .. code-block:: bash - tutor dev run lms ./manage.py lms cache_programs --domain="local.edly.io:8000" + tutor dev run lms ./manage.py lms cache_programs --domain="local.openedx.io:8000" This last step should be performed every time you create new or make changes to existing programs. @@ -122,7 +122,7 @@ Show Programs Tab ~~~~~~~~~~~~~~~~~ To make the ``Programs`` tab work in the LMS dashboard, users will need to manually create an entry -in the ``Programs api config`` model in the LMS Admin Panel. Go to http://local.edly.io/admin/programs/programsapiconfig/. +in the ``Programs api config`` model in the LMS Admin Panel. Go to http://local.openedx.io/admin/programs/programsapiconfig/. Add ``Marketing path`` equal to ``/programs`` and enable it. Then Programs tab will be shown on the LMS where users can view their registered programs. It will show like in the below picture. @@ -132,7 +132,7 @@ where users can view their registered programs. It will show like in the below p In the above image, the user can see explore programs button which is pointing to ``http://localhost:8080/programs`` by default. This link does not exist. So, users can change this link to their custom-built marketing site URL to show all programs there. This can be done by modifying the ``Site Configurations`` model in the LMS Admin Panel. Go to -http://local.edly.io/admin/site_configuration/siteconfiguration/. Open the respective LMS site configuration and add the below +http://local.openedx.io/admin/site_configuration/siteconfiguration/. Open the respective LMS site configuration and add the below dictionary in ``site values`` field like the below image: .. code-block:: python @@ -169,7 +169,7 @@ in the development container: tutor dev start --mount /path/to/course-discovery/ discovery -You can then access the development server at http://discovery.local.edly.io:8381. Feel free to add breakpoints +You can then access the development server at http://discovery.local.openedx.io:8381. Feel free to add breakpoints (``import pdb; pdb.set_trace()``) anywhere in your source code to debug your application. Once a local repository is mounted in the image, you will have to install nodejs dependencies and collect static assets: From 1cc9c3f40c191bb28031797ed70a67114cdbba69 Mon Sep 17 00:00:00 2001 From: Hina Khadim Date: Wed, 23 Oct 2024 16:43:16 +0500 Subject: [PATCH 2/6] feat: ubuntu upgrade in dockerfile (#84) * feat: ubuntu upgrade in dockerfile * fix: set app id to 1000 * fix: remove uwsgi * fix: remove empty lines * fix: update changelog --- ...41018_185457_hina.khadim_upgrade_ubuntu.md | 1 + .../discovery/build/discovery/Dockerfile | 24 +++++++++---------- 2 files changed, 13 insertions(+), 12 deletions(-) create mode 100644 changelog.d/20241018_185457_hina.khadim_upgrade_ubuntu.md diff --git a/changelog.d/20241018_185457_hina.khadim_upgrade_ubuntu.md b/changelog.d/20241018_185457_hina.khadim_upgrade_ubuntu.md new file mode 100644 index 0000000..13eb288 --- /dev/null +++ b/changelog.d/20241018_185457_hina.khadim_upgrade_ubuntu.md @@ -0,0 +1 @@ +- 💥[Feature] Update Course Discovery Image to use Ubuntu 22.04 as base OS. (by @hinakhadim) diff --git a/tutordiscovery/templates/discovery/build/discovery/Dockerfile b/tutordiscovery/templates/discovery/build/discovery/Dockerfile index 87894cd..cbe21a8 100644 --- a/tutordiscovery/templates/discovery/build/discovery/Dockerfile +++ b/tutordiscovery/templates/discovery/build/discovery/Dockerfile @@ -1,6 +1,6 @@ # syntax=docker/dockerfile:1.4 ###### Minimal image with base system requirements for most stages -FROM docker.io/ubuntu:20.04 AS minimal +FROM docker.io/ubuntu:22.04 AS minimal ENV DEBIAN_FRONTEND=noninteractive RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ @@ -8,7 +8,7 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ apt update && \ apt install -y curl git-core gettext language-pack-en \ build-essential libcairo2 libffi-dev libmysqlclient-dev libxml2-dev libxslt-dev libjpeg-dev libssl-dev \ - pkg-config libsqlite3-dev mime-support + pkg-config libsqlite3-dev media-types mailcap ENV LC_ALL=en_US.UTF-8 ARG APP_USER_ID=1000 @@ -91,13 +91,13 @@ RUN mkdir course_discovery/media # Run production server ENV DJANGO_SETTINGS_MODULE=course_discovery.settings.tutor.production EXPOSE 8000 -CMD uwsgi \ - --static-map /static=/openedx/discovery/course_discovery/assets \ - --static-map /media=/openedx/discovery/course_discovery/media \ - --http 0.0.0.0:8000 \ - --thunder-lock \ - --single-interpreter \ - --enable-threads \ - --processes=2 \ - --buffer-size=8192 \ - --wsgi-file course_discovery/wsgi.py +CMD ["uwsgi", \ + "--static-map", "/static=/openedx/discovery/course_discovery/assets", \ + "--static-map", "/media=/openedx/discovery/course_discovery/media", \ + "--http", "0.0.0.0:8000", \ + "--thunder-lock", \ + "--single-interpreter", \ + "--enable-threads", \ + "--processes=2", \ + "--buffer-size=8192", \ + "--wsgi-file", "course_discovery/wsgi.py"] From b89af4240630f35a6a76ae0b2bf80f7a5fdd4536 Mon Sep 17 00:00:00 2001 From: Muhammad Faraz Maqsood Date: Fri, 15 Nov 2024 12:48:21 +0500 Subject: [PATCH 3/6] v19.0.0 upgrade to sumac --- CHANGELOG.md | 10 ++++++++++ changelog.d/20240621_170044_regis.md | 1 - ...26_115731_faraz.maqsood_images_persistance_issue.md | 1 - ...247_evgen.dyudyunov_fix_service_user_permissions.md | 1 - .../20241018_185457_hina.khadim_upgrade_ubuntu.md | 1 - setup.py | 4 ++-- tutordiscovery/__about__.py | 2 +- .../templates/discovery/build/discovery/Dockerfile | 6 +++--- 8 files changed, 16 insertions(+), 10 deletions(-) delete mode 100644 changelog.d/20240621_170044_regis.md delete mode 100644 changelog.d/20240626_115731_faraz.maqsood_images_persistance_issue.md delete mode 100644 changelog.d/20240710_160247_evgen.dyudyunov_fix_service_user_permissions.md delete mode 100644 changelog.d/20241018_185457_hina.khadim_upgrade_ubuntu.md diff --git a/CHANGELOG.md b/CHANGELOG.md index faf3be1..6090f6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,16 @@ instructions, because git commits are used to generate release notes: + +## v19.0.0 (2024-10-23) + +- 💥[Feature] Upgrade to Sumac. (by @Faraz32123) +- 💥[Feature] Update Course Discovery Image to use Ubuntu 22.04 as base OS. (by @hinakhadim) +- [Bugfix] Fix catalog_service_user permissions and 403 while fetching pathways (by @dyudyunov) +- [BugFix] Fix images(media) persistance issue by mounting media directory in volumes through patches. (by @Faraz32123) +- [Bugfix] Fix legacy warnings during Docker build. (by @regisb) + + ## v18.0.0 (2024-05-14) diff --git a/changelog.d/20240621_170044_regis.md b/changelog.d/20240621_170044_regis.md deleted file mode 100644 index d7d2eb7..0000000 --- a/changelog.d/20240621_170044_regis.md +++ /dev/null @@ -1 +0,0 @@ -- [Bugfix] Fix legacy warnings during Docker build. (by @regisb) diff --git a/changelog.d/20240626_115731_faraz.maqsood_images_persistance_issue.md b/changelog.d/20240626_115731_faraz.maqsood_images_persistance_issue.md deleted file mode 100644 index 02233c5..0000000 --- a/changelog.d/20240626_115731_faraz.maqsood_images_persistance_issue.md +++ /dev/null @@ -1 +0,0 @@ -- [BugFix] Fix images(media) persistance issue by mounting media directory in volumes through patches. (by @Faraz32123) diff --git a/changelog.d/20240710_160247_evgen.dyudyunov_fix_service_user_permissions.md b/changelog.d/20240710_160247_evgen.dyudyunov_fix_service_user_permissions.md deleted file mode 100644 index 2b5adb9..0000000 --- a/changelog.d/20240710_160247_evgen.dyudyunov_fix_service_user_permissions.md +++ /dev/null @@ -1 +0,0 @@ -- [Bugfix] Fix catalog_service_user permissions and 403 while fetching pathways (by @dyudyunov) diff --git a/changelog.d/20241018_185457_hina.khadim_upgrade_ubuntu.md b/changelog.d/20241018_185457_hina.khadim_upgrade_ubuntu.md deleted file mode 100644 index 13eb288..0000000 --- a/changelog.d/20241018_185457_hina.khadim_upgrade_ubuntu.md +++ /dev/null @@ -1 +0,0 @@ -- 💥[Feature] Update Course Discovery Image to use Ubuntu 22.04 as base OS. (by @hinakhadim) diff --git a/setup.py b/setup.py index 7997045..30f574c 100644 --- a/setup.py +++ b/setup.py @@ -34,8 +34,8 @@ long_description_content_type="text/x-rst", packages=find_packages(exclude=["tests*"]), include_package_data=True, - install_requires=["tutor>=18.0.0,<19.0.0"], - extras_require={"dev": "tutor[dev]>=18.0.0,<19.0.0"}, + install_requires=["tutor>=19.0.0,<20.0.0"], + extras_require={"dev": "tutor[dev]>=19.0.0,<20.0.0"}, python_requires=">=3.9", entry_points={"tutor.plugin.v1": ["discovery = tutordiscovery.plugin"]}, classifiers=[ diff --git a/tutordiscovery/__about__.py b/tutordiscovery/__about__.py index c6a8b8e..0122a6f 100644 --- a/tutordiscovery/__about__.py +++ b/tutordiscovery/__about__.py @@ -1 +1 @@ -__version__ = "18.0.0" +__version__ = "19.0.0" diff --git a/tutordiscovery/templates/discovery/build/discovery/Dockerfile b/tutordiscovery/templates/discovery/build/discovery/Dockerfile index cbe21a8..ef9016e 100644 --- a/tutordiscovery/templates/discovery/build/discovery/Dockerfile +++ b/tutordiscovery/templates/discovery/build/discovery/Dockerfile @@ -47,10 +47,10 @@ RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared pip install \ # https://pypi.org/project/setuptools/ # https://pypi.org/project/pip/ # https://pypi.org/project/wheel/ - setuptools==69.1.1 pip==24.0 wheel==0.43.0 + setuptools==75.2.0 pip==24.2 wheel==0.44.0 # Install a recent version of nodejs -RUN pip install nodeenv==1.8.0 +RUN pip install nodeenv==1.9.1 # nodejs version picked from https://github.com/openedx/course-discovery/blob/master/Dockerfile RUN nodeenv /openedx/nodeenv --node=16.14.2 --prebuilt ENV PATH=/openedx/nodeenv/bin:${PATH} @@ -73,7 +73,7 @@ RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared pip install \ # Use redis as a django cache https://pypi.org/project/django-redis/ django-redis==5.4.0 \ # uwsgi server https://pypi.org/project/uWSGI/ - uwsgi==2.0.24 + uwsgi==2.0.27 {% if DISCOVERY_ATLAS_PULL %} # Pull translations. Support the OEP-58 proposal behind a feature flag until it's fully implemented. From 403a5e297d7ed1489686d84ac174a9902faa0b0d Mon Sep 17 00:00:00 2001 From: Muhammad Faraz Maqsood Date: Fri, 1 Nov 2024 22:22:22 +0500 Subject: [PATCH 4/6] feat: add Elasticsearch support in tutor-discovery (#89) - As Tutor and Open edX have shifted to Meilisearch, and course-discovery still depends on Elasticsearch, running the Elasticsearch container with tutor-discovery will facilitate smoother operation for the course-discovery service. - It's related PR from tutor: overhangio/tutor#1141. Co-authored-by: Muhammad Faraz Maqsood --- ...ood_enable_elastic_search_for_discovery.md | 2 + .../patches/discovery-common-settings | 1 + tutordiscovery/patches/k8s-deployments | 51 +++++++++++++++++++ tutordiscovery/patches/k8s-jobs | 1 - tutordiscovery/patches/k8s-services | 16 ++++++ tutordiscovery/patches/k8s-volumes | 16 ++++++ .../patches/local-docker-compose-dev-services | 9 ++++ .../local-docker-compose-permissions-command | 1 + .../local-docker-compose-permissions-volumes | 1 + .../patches/local-docker-compose-services | 23 ++++++++- tutordiscovery/plugin.py | 6 +++ .../apps/settings/partials/common.py | 8 ++- 12 files changed, 132 insertions(+), 3 deletions(-) create mode 100644 changelog.d/20241025_181736_faraz.maqsood_enable_elastic_search_for_discovery.md create mode 100644 tutordiscovery/patches/discovery-common-settings create mode 100644 tutordiscovery/patches/k8s-volumes create mode 100644 tutordiscovery/patches/local-docker-compose-permissions-command create mode 100644 tutordiscovery/patches/local-docker-compose-permissions-volumes diff --git a/changelog.d/20241025_181736_faraz.maqsood_enable_elastic_search_for_discovery.md b/changelog.d/20241025_181736_faraz.maqsood_enable_elastic_search_for_discovery.md new file mode 100644 index 0000000..551ea27 --- /dev/null +++ b/changelog.d/20241025_181736_faraz.maqsood_enable_elastic_search_for_discovery.md @@ -0,0 +1,2 @@ +- [Feature] Add Elasticsearch support in tutor-discovery. As Tutor and Open edX have shifted to Meilisearch, and course-discovery still depends on Elasticsearch, running the Elasticsearch container with tutor-discovery will facilitate smoother operation for the course-discovery service. (by @Faraz32123) + - It's related PR from tutor: https://github.com/overhangio/tutor/pull/1141. diff --git a/tutordiscovery/patches/discovery-common-settings b/tutordiscovery/patches/discovery-common-settings new file mode 100644 index 0000000..8bc08f1 --- /dev/null +++ b/tutordiscovery/patches/discovery-common-settings @@ -0,0 +1 @@ +DISCOVERY_RUN_ELASTICSEARCH = {{ DISCOVERY_RUN_ELASTICSEARCH }} diff --git a/tutordiscovery/patches/k8s-deployments b/tutordiscovery/patches/k8s-deployments index 15a1b3f..06e959c 100644 --- a/tutordiscovery/patches/k8s-deployments +++ b/tutordiscovery/patches/k8s-deployments @@ -32,3 +32,54 @@ spec: - name: settings configMap: name: discovery-settings + +{% if DISCOVERY_RUN_ELASTICSEARCH %} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: elasticsearch + labels: + app.kubernetes.io/name: elasticsearch +spec: + selector: + matchLabels: + app.kubernetes.io/name: elasticsearch + strategy: + type: Recreate + template: + metadata: + labels: + app.kubernetes.io/name: elasticsearch + spec: + securityContext: + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + fsGroupChangePolicy: "OnRootMismatch" + containers: + - name: elasticsearch + image: {{ DISCOVERY_DOCKER_IMAGE_ELASTICSEARCH }} + env: + - name: cluster.name + value: "openedx" + - name: bootstrap.memory_lock + value: "true" + - name: discovery.type + value: "single-node" + - name: ES_JAVA_OPTS + value: "-Xms{{ DISCOVERY_ELASTICSEARCH_HEAP_SIZE }} -Xmx{{ DISCOVERY_ELASTICSEARCH_HEAP_SIZE }}" + - name: TAKE_FILE_OWNERSHIP + value: "1" + ports: + - containerPort: 9200 + securityContext: + allowPrivilegeEscalation: false + volumeMounts: + - mountPath: /usr/share/elasticsearch/data + name: data + volumes: + - name: data + persistentVolumeClaim: + claimName: elasticsearch +{% endif %} diff --git a/tutordiscovery/patches/k8s-jobs b/tutordiscovery/patches/k8s-jobs index d45e8d7..b3409ab 100644 --- a/tutordiscovery/patches/k8s-jobs +++ b/tutordiscovery/patches/k8s-jobs @@ -23,4 +23,3 @@ spec: - name: settings configMap: name: discovery-settings - diff --git a/tutordiscovery/patches/k8s-services b/tutordiscovery/patches/k8s-services index feb9752..3592338 100644 --- a/tutordiscovery/patches/k8s-services +++ b/tutordiscovery/patches/k8s-services @@ -10,3 +10,19 @@ spec: protocol: TCP selector: app.kubernetes.io/name: discovery +{% if DISCOVERY_RUN_ELASTICSEARCH %} +--- +apiVersion: v1 +kind: Service +metadata: + name: elasticsearch + labels: + app.kubernetes.io/name: elasticsearch +spec: + type: ClusterIP + ports: + - port: 9200 + protocol: TCP + selector: + app.kubernetes.io/name: elasticsearch +{% endif %} diff --git a/tutordiscovery/patches/k8s-volumes b/tutordiscovery/patches/k8s-volumes new file mode 100644 index 0000000..4f3ba6b --- /dev/null +++ b/tutordiscovery/patches/k8s-volumes @@ -0,0 +1,16 @@ +{% if DISCOVERY_RUN_ELASTICSEARCH %} +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: elasticsearch + labels: + app.kubernetes.io/component: volume + app.kubernetes.io/name: elasticsearch +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 2Gi +{% endif %} diff --git a/tutordiscovery/patches/local-docker-compose-dev-services b/tutordiscovery/patches/local-docker-compose-dev-services index 95906db..43356e7 100644 --- a/tutordiscovery/patches/local-docker-compose-dev-services +++ b/tutordiscovery/patches/local-docker-compose-dev-services @@ -15,3 +15,12 @@ discovery: default: aliases: - "{{ DISCOVERY_HOST }}" + +{% if DISCOVERY_RUN_ELASTICSEARCH and is_docker_rootless() %} + elasticsearch: + ulimits: + memlock: + # Fixes error setting rlimits for ready process in rootless docker + soft: 0 # zero means "unset" in the memlock context + hard: 0 + {% endif %} diff --git a/tutordiscovery/patches/local-docker-compose-permissions-command b/tutordiscovery/patches/local-docker-compose-permissions-command new file mode 100644 index 0000000..9f470cd --- /dev/null +++ b/tutordiscovery/patches/local-docker-compose-permissions-command @@ -0,0 +1 @@ +{% if DISCOVERY_RUN_ELASTICSEARCH %}setowner 1000 /mounts/elasticsearch{% endif %} diff --git a/tutordiscovery/patches/local-docker-compose-permissions-volumes b/tutordiscovery/patches/local-docker-compose-permissions-volumes new file mode 100644 index 0000000..5bae503 --- /dev/null +++ b/tutordiscovery/patches/local-docker-compose-permissions-volumes @@ -0,0 +1 @@ +{% if DISCOVERY_RUN_ELASTICSEARCH %}- ../../data/elasticsearch:/mounts/elasticsearch{% endif %} diff --git a/tutordiscovery/patches/local-docker-compose-services b/tutordiscovery/patches/local-docker-compose-services index ddbf6ab..bd1bd57 100644 --- a/tutordiscovery/patches/local-docker-compose-services +++ b/tutordiscovery/patches/local-docker-compose-services @@ -6,7 +6,28 @@ discovery: volumes: - ../plugins/discovery/apps/settings/tutor:/openedx/discovery/course_discovery/settings/tutor:ro - ../../data/discovery/media:/openedx/discovery/course_discovery/media + {% if DISCOVERY_RUN_ELASTICSEARCH %}- ../../data/elasticsearch:/mounts/elasticsearch{% endif %} depends_on: - lms {% if RUN_MYSQL %}- mysql{% endif %} - {% if RUN_ELASTICSEARCH %}- elasticsearch{% endif %} + {% if DISCOVERY_RUN_ELASTICSEARCH %}- elasticsearch{% endif %} + +{% if DISCOVERY_RUN_ELASTICSEARCH -%} + elasticsearch: + image: {{ DISCOVERY_DOCKER_IMAGE_ELASTICSEARCH }} + environment: + - cluster.name=openedx + - bootstrap.memory_lock=true + - discovery.type=single-node + - "ES_JAVA_OPTS=-Xms{{ DISCOVERY_ELASTICSEARCH_HEAP_SIZE }} -Xmx{{ DISCOVERY_ELASTICSEARCH_HEAP_SIZE }}" + ulimits: + memlock: + soft: -1 + hard: -1 + restart: unless-stopped + user: "1000:1000" + volumes: + - ../../data/elasticsearch:/usr/share/elasticsearch/data + depends_on: + - permissions +{%- endif %} diff --git a/tutordiscovery/plugin.py b/tutordiscovery/plugin.py index 6633d5d..9546569 100644 --- a/tutordiscovery/plugin.py +++ b/tutordiscovery/plugin.py @@ -36,6 +36,12 @@ "EXTRA_PIP_REQUIREMENTS": [], "REPOSITORY": "https://github.com/openedx/course-discovery.git", "REPOSITORY_VERSION": "{{ OPENEDX_COMMON_VERSION }}", + "RUN_ELASTICSEARCH": True, + "DOCKER_IMAGE_ELASTICSEARCH": "docker.io/elasticsearch:7.17.13", + "ELASTICSEARCH_HOST": "elasticsearch", + "ELASTICSEARCH_PORT": 9200, + "ELASTICSEARCH_SCHEME": "http", + "ELASTICSEARCH_HEAP_SIZE": "1g", }, "unique": { "MYSQL_PASSWORD": "{{ 8|random_string }}", diff --git a/tutordiscovery/templates/discovery/apps/settings/partials/common.py b/tutordiscovery/templates/discovery/apps/settings/partials/common.py index 4cd4758..424639a 100644 --- a/tutordiscovery/templates/discovery/apps/settings/partials/common.py +++ b/tutordiscovery/templates/discovery/apps/settings/partials/common.py @@ -20,8 +20,14 @@ } } +DISCOVERY_DOCKER_IMAGE_ELASTICSEARCH = "{{ DISCOVERY_DOCKER_IMAGE_ELASTICSEARCH }}" +DISCOVERY_ELASTICSEARCH_HOST = "{{ DISCOVERY_ELASTICSEARCH_HOST }}" +DISCOVERY_ELASTICSEARCH_PORT = "{{ DISCOVERY_ELASTICSEARCH_PORT }}" +DISCOVERY_ELASTICSEARCH_SCHEME = "{{ DISCOVERY_ELASTICSEARCH_SCHEME }}" +DISCOVERY_ELASTICSEARCH_HEAP_SIZE = "{{ DISCOVERY_ELASTICSEARCH_HEAP_SIZE }}" + ELASTICSEARCH_DSL['default'].update({ - 'hosts': "{{ ELASTICSEARCH_SCHEME }}://{{ ELASTICSEARCH_HOST }}:{{ ELASTICSEARCH_PORT }}/" + 'hosts': "{{ DISCOVERY_ELASTICSEARCH_SCHEME }}://{{ DISCOVERY_ELASTICSEARCH_HOST }}:{{ DISCOVERY_ELASTICSEARCH_PORT }}/" }) {% for name, index in DISCOVERY_INDEX_OVERRIDES.items() %} From b05698b9a3bcef8a2eb9923a56deedecdac3de6c Mon Sep 17 00:00:00 2001 From: Muhammad Faraz Maqsood Date: Fri, 22 Nov 2024 12:44:59 +0500 Subject: [PATCH 5/6] chore: add is_docker_rootless() in tutor-discovery - move is_docker_rootless method from tutor to tutor-discovery - move is_docker_rootless related tests from tutor to tutor-discovery and modify makefile according to it. --- Makefile | 7 +++-- .../20241122_121506_faraz.maqsood_sumac.md | 1 + tests/__init__.py | 0 tests/test_utils.py | 26 +++++++++++++++++++ tutordiscovery/plugin.py | 5 ++++ tutordiscovery/utils.py | 16 ++++++++++++ 6 files changed, 53 insertions(+), 2 deletions(-) create mode 100644 changelog.d/20241122_121506_faraz.maqsood_sumac.md create mode 100644 tests/__init__.py create mode 100644 tests/test_utils.py create mode 100644 tutordiscovery/utils.py diff --git a/Makefile b/Makefile index 2adc5ab..5bda776 100644 --- a/Makefile +++ b/Makefile @@ -1,10 +1,10 @@ .DEFAULT_GOAL := help .PHONY: docs -SRC_DIRS = ./tutordiscovery +SRC_DIRS = ./tutordiscovery ./tests BLACK_OPTS = --exclude templates ${SRC_DIRS} # Warning: These checks are run on every PR. -test: test-lint test-types test-format # Run some static checks. +test: test-lint test-types test-format test-unit # Run some static checks. test-format: ## Run code formatting tests. black --check --diff $(BLACK_OPTS) @@ -15,6 +15,9 @@ test-lint: ## Run code linting tests test-types: ## Run type checks. mypy --exclude=templates --ignore-missing-imports --implicit-reexport --strict ${SRC_DIRS} +test-unit: ## Run unit tests + python -m unittest discover tests + format: ## Format code automatically. black $(BLACK_OPTS) diff --git a/changelog.d/20241122_121506_faraz.maqsood_sumac.md b/changelog.d/20241122_121506_faraz.maqsood_sumac.md new file mode 100644 index 0000000..3c77ef5 --- /dev/null +++ b/changelog.d/20241122_121506_faraz.maqsood_sumac.md @@ -0,0 +1 @@ +- [Improvement] Move is_docker_rootless method related to elasticsearch from tutor core to tutor-discovery. (by @Faraz32123) diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_utils.py b/tests/test_utils.py new file mode 100644 index 0000000..0a2f766 --- /dev/null +++ b/tests/test_utils.py @@ -0,0 +1,26 @@ +import subprocess +import unittest +from unittest.mock import MagicMock, patch + +from tutordiscovery import utils + + +class UtilsTests(unittest.TestCase): + @patch("subprocess.run") + def test_is_docker_rootless(self, mock_run: MagicMock) -> None: + # Mock rootless `docker info` output + utils.is_docker_rootless.cache_clear() + mock_run.return_value.stdout = "some prefix\n rootless foo bar".encode("utf-8") + self.assertTrue(utils.is_docker_rootless()) + + # Mock regular `docker info` output + utils.is_docker_rootless.cache_clear() + mock_run.return_value.stdout = "some prefix, regular docker".encode("utf-8") + self.assertFalse(utils.is_docker_rootless()) + + @patch("subprocess.run") + def test_is_docker_rootless_podman(self, mock_run: MagicMock) -> None: + """Test the `is_docker_rootless` when podman is used or any other error with `docker info`""" + utils.is_docker_rootless.cache_clear() + mock_run.side_effect = subprocess.CalledProcessError(1, "docker info") + self.assertFalse(utils.is_docker_rootless()) diff --git a/tutordiscovery/plugin.py b/tutordiscovery/plugin.py index 9546569..bb47323 100644 --- a/tutordiscovery/plugin.py +++ b/tutordiscovery/plugin.py @@ -9,6 +9,7 @@ from tutor.__about__ import __version_suffix__ from .__about__ import __version__ +from .utils import is_docker_rootless # Handle version suffix in main mode, just like tutor core if __version_suffix__: @@ -127,6 +128,10 @@ def _mount_course_discovery_on_build( ("discovery/apps", "plugins"), ], ) +# Template variables +tutor_hooks.Filters.ENV_TEMPLATE_VARIABLES.add_item( + ("is_docker_rootless", is_docker_rootless), +) # Load patches from files for path in glob( os.path.join( diff --git a/tutordiscovery/utils.py b/tutordiscovery/utils.py new file mode 100644 index 0000000..cfab7f3 --- /dev/null +++ b/tutordiscovery/utils.py @@ -0,0 +1,16 @@ +import subprocess +from functools import lru_cache + + +@lru_cache(maxsize=None) +def is_docker_rootless() -> bool: + """ + A helper function to determine if Docker is running in rootless mode. + + - https://docs.docker.com/engine/security/rootless/ + """ + try: + results = subprocess.run(["docker", "info"], capture_output=True, check=True) + return "rootless" in results.stdout.decode() + except subprocess.CalledProcessError: + return False From 214ae646289fff268116c9b9c6e5f45b2af6806d Mon Sep 17 00:00:00 2001 From: Syed Muhammad Dawoud Sheraz Ali Date: Mon, 9 Dec 2024 19:07:49 +0500 Subject: [PATCH 6/6] chore: update changelog --- CHANGELOG.md | 7 +++++++ ...36_faraz.maqsood_enable_elastic_search_for_discovery.md | 2 -- ...20241111_172041_faraz.maqsood_remove_py38_references.md | 1 - changelog.d/20241119_123431_dawoud.sheraz_branch_rename.md | 3 --- changelog.d/20241122_121506_faraz.maqsood_sumac.md | 1 - 5 files changed, 7 insertions(+), 7 deletions(-) delete mode 100644 changelog.d/20241025_181736_faraz.maqsood_enable_elastic_search_for_discovery.md delete mode 100644 changelog.d/20241111_172041_faraz.maqsood_remove_py38_references.md delete mode 100644 changelog.d/20241119_123431_dawoud.sheraz_branch_rename.md delete mode 100644 changelog.d/20241122_121506_faraz.maqsood_sumac.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 6090f6e..796dc4a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,13 @@ instructions, because git commits are used to generate release notes: ## v19.0.0 (2024-10-23) - 💥[Feature] Upgrade to Sumac. (by @Faraz32123) +- [Feature] Add Elasticsearch support in tutor-discovery. As Tutor and Open edX have shifted to Meilisearch, and course-discovery still depends on Elasticsearch, running the Elasticsearch container with tutor-discovery will facilitate smoother operation for the course-discovery service. (by @Faraz32123) + - Please see related tutor core PR for context https://github.com/overhangio/tutor/pull/1141 +- 💥 [Deprecation] Drop support for python 3.8 and set Python 3.9 as the minimum supported python version. (by @Faraz32123) +- 💥[Improvement] Rename Tutor's two branches (by @DawoudSheraz): + * Rename **master** to **release**, as this branch runs the latest official Open edX release tag. + * Rename **nightly** to **main**, as this branch runs the Open edX master branches, which are the basis for the next Open edX release. +- [Improvement] Move is_docker_rootless method related to elasticsearch from tutor core to tutor-discovery. (by @Faraz32123) - 💥[Feature] Update Course Discovery Image to use Ubuntu 22.04 as base OS. (by @hinakhadim) - [Bugfix] Fix catalog_service_user permissions and 403 while fetching pathways (by @dyudyunov) - [BugFix] Fix images(media) persistance issue by mounting media directory in volumes through patches. (by @Faraz32123) diff --git a/changelog.d/20241025_181736_faraz.maqsood_enable_elastic_search_for_discovery.md b/changelog.d/20241025_181736_faraz.maqsood_enable_elastic_search_for_discovery.md deleted file mode 100644 index 551ea27..0000000 --- a/changelog.d/20241025_181736_faraz.maqsood_enable_elastic_search_for_discovery.md +++ /dev/null @@ -1,2 +0,0 @@ -- [Feature] Add Elasticsearch support in tutor-discovery. As Tutor and Open edX have shifted to Meilisearch, and course-discovery still depends on Elasticsearch, running the Elasticsearch container with tutor-discovery will facilitate smoother operation for the course-discovery service. (by @Faraz32123) - - It's related PR from tutor: https://github.com/overhangio/tutor/pull/1141. diff --git a/changelog.d/20241111_172041_faraz.maqsood_remove_py38_references.md b/changelog.d/20241111_172041_faraz.maqsood_remove_py38_references.md deleted file mode 100644 index 8b70c7b..0000000 --- a/changelog.d/20241111_172041_faraz.maqsood_remove_py38_references.md +++ /dev/null @@ -1 +0,0 @@ -- 💥 [Deprecation] Drop support for python 3.8 and set Python 3.9 as the minimum supported python version. (by @Faraz32123) diff --git a/changelog.d/20241119_123431_dawoud.sheraz_branch_rename.md b/changelog.d/20241119_123431_dawoud.sheraz_branch_rename.md deleted file mode 100644 index fb6e264..0000000 --- a/changelog.d/20241119_123431_dawoud.sheraz_branch_rename.md +++ /dev/null @@ -1,3 +0,0 @@ -- 💥[Improvement] Rename Tutor's two branches (by @DawoudSheraz): - * Rename **master** to **release**, as this branch runs the latest official Open edX release tag. - * Rename **nightly** to **main**, as this branch runs the Open edX master branches, which are the basis for the next Open edX release. \ No newline at end of file diff --git a/changelog.d/20241122_121506_faraz.maqsood_sumac.md b/changelog.d/20241122_121506_faraz.maqsood_sumac.md deleted file mode 100644 index 3c77ef5..0000000 --- a/changelog.d/20241122_121506_faraz.maqsood_sumac.md +++ /dev/null @@ -1 +0,0 @@ -- [Improvement] Move is_docker_rootless method related to elasticsearch from tutor core to tutor-discovery. (by @Faraz32123)