MST-1789 downstream triggers (#191)

* feat: add openedx_events and edx_event_bus_kafka to INSTALLED_APPS * feat: install confluent_kafka[avro,schema-registry] for use with edx-event-bus-kafka * feat: add event bus and Kafka Django settings * feat: make User.full_name field non-nullable This commit makes the User.full_name field non-nullable to ensure that there is only one representation of the empty value - the empty string. Currently, empty values for this field can be either None or the empty string. Because we're transitioning from a nullable to non-nullable state, a default is required to handle existing rows with a null value. This is best practice; Django discourages setting both null=True and blank=True on CharField model fields. Furthermore, this was required by our event bus work. If an empty value is represented by None, this causes issues with the event bus, because None is not JSON serializable. Instead of converting a None value to the empty string in the event producer, correcting the the model definition is a better approach. * feat: upgrade openedx-events to 8.8.0 * feat: emit the EXAM_ATTEMPT_SUBMITTED Open edX event when an exam is submitted * feat: emit verified event for attempt (#186) * feat: emit rejected exam signal (#181) * feat: add errored event producer (#189) * chore: fix upgrades * chore: remove confluent-kafka import * docs: local event bus development * fix: remove local Kafka settings This commit removes Django settings from the local settings file, used by the local application server, related to setting up the Kafka implementation of the event bus. This is because the event bus does not work outside of a Docker container. This is because the event bus is run through the devstack networking layer, which is inaccessible by the local application server. * feat: update topic names for events (#193) * feat: exam reset producer (#196) send event to the event bus when an exam attempt is reset --------- Co-authored-by: michaelroytman <[email protected]> Co-authored-by: Michael Roytman <[email protected]> Co-authored-by: Zachary Hancock <[email protected]>
edx · Oct 6, 2023 · 0756267 · 0756267
1 parent b8c5804
commit 0756267
Show file tree

Hide file tree

Showing 29 changed files with 678 additions and 320 deletions.
diff --git a/.coveragerc b/.coveragerc
@@ -6,6 +6,7 @@ omit =
     edx_exams/settings/*
     edx_exams/conf*
     edx_exams/docker_gunicorn_configuration.py
+    edx_exams/apps/core/signals/handlers.py
     *wsgi.py
     *migrations*
     *admin.py

diff --git a/README.rst b/README.rst
@@ -20,8 +20,8 @@ Documentation
 Development Workflow
 --------------------
 
-One Time Setup
-~~~~~~~~~~~~~~
+Local Development Set Up
+~~~~~~~~~~~~~~~~~~~~~~~~
 .. code-block::
 
   # Clone the repository
@@ -43,9 +43,24 @@ One Time Setup
   # Run edx-exams locally
   python manage.py runserver localhost:18740 --settings=edx_exams.settings.local
 
+Devstack Set Up
+~~~~~~~~~~~~~~~
+.. code-block::
+
+  # Clone the repository
+  git clone [email protected]:edx/edx-exams.git
+  cd edx-exams
 
-Every time you develop something in this repo
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+  # Start LMS in devstack from your local devstack directory
+  make dev.up.lms
+
+  # Return to the edx-exams repo directory and provision the edx-exams containers
+  bash provision-edx-exams.sh
+
+You can use the make targets defined in the ``Makefile`` to interact with the running ``edx-exams`` Docker containers.
+
+Development Workflow
+~~~~~~~~~~~~~~~~~~~~
 .. code-block::
 
   # Activate the virtualenv
@@ -79,6 +94,88 @@ Every time you develop something in this repo
 
   # Open a PR and ask for review.
 
+Event Bus Set Up
+~~~~~~~~~~~~~~~~
+
+The ``edx-exams`` service uses the Open edX event bus to publish events relating to the exam attempt lifecycle and
+others important exam events. These Open edX events are emitted by the service and pushed onto the event bus. Downstream
+services, like the LMS, receive these events and implement downstream effects of these events. For more details,
+please see `Implementation of Event Driven Architecture for Exam Downstream Effects`_.
+
+These focus of these instructions is on how to set up the Open edX event bus for use with ``edx-exams``. For more
+documentation about the event bus in general, please see `How to start using the Event Bus`_.
+
+Currently, the event bus is only supported in environments running Docker containers, like `devstack`_. This is because
+the interactions between services on the event bus is implemented in the devstack networking layer.
+
+In order to run the event bus locally, follow these steps. These steps assume that you both have `devstack`_ running and
+that you are running the ``edx-exams`` Docker container, as described in the Devstack Set Up section. These steps
+describe how to install and run the Kafka-based event bus.
+
+1. In a ``requirements/private.txt`` file, add the following Python package. These requirements are necessary for the
+   Kafka-based event bus. They are not included as a part of the standard set of requirements because installation of
+   confluent_kafka poses issues for users of Tutor on M1 Macs, which includes many users in the Open edX community. 
+   For more details, please see `Optional Import of Confluent Kafka`_.
+
+
+  .. code-block::
+
+    confluent_kafka[avro,schema-registry]
+
+2. Install the application requirements to install ``confluent_kafka``.
+
+  .. code-block::
+
+    # Shell into the application Docker container
+    make app-shell
+
+    # Install requirements
+    make requirements
+
+3. Follow the `manual testing`_ instructions to set up the Kafka-based Open edX event bus in the service that contains
+   the event handler(s) for your event(s) - for example, the LMS or Studio.
+
+Producing Events
+################
+
+Events will be produced at key stages of the exam attempt lifecycle and other points in the special exam feature. If you
+are using the local Kafka cluster, you will be able to see the topics and events there.
+
+Consuming Events
+################
+
+In order to consume events off the event bus, you must run a management command that starts an infinite loop to read
+from the event bus.
+
+Shell into the application Docker container and run the following management command to start the loop. See the
+`consume_events management command documentation`_ for a description of the arguments.
+
+.. code-block::
+
+  python3 manage.py consume_events -t <topic-name> -g <group-id>
+
+Here is an example of a command to consume events from the ``learning-exam-attempt-lifecycle`` topic in the LMS.
+
+.. code-block::
+
+    python3 manage.py ls consume_events -t learning-exam-attempt-lifecycle -g dev-lms
+
+When your event is successfully consumed, you should see logs like the following.
+
+.. code-block::
+
+  2023-10-04 15:50:17,508 INFO 554 [edx_event_bus_kafka.internal.consumer] [user None] [ip None] consumer.py:513 - Message received from Kafka: topic=dev-learning-exam-attempt-lifecycle, partition=0, offset=7, message_id=b71c735c-62cd-11ee-9064-0242ac120012, key=b'\x00\x00\x00\x00\x010course-v1:edX+777+2023FW', event_timestamp_ms=1696434617498
+
+  2023-10-04 15:50:17,593 INFO 554 [edx_event_bus_kafka.internal.consumer] [user None] [ip None] consumer.py:393 - Message from Kafka processed successfully
+
+
+.. _Implementation of Event Driven Architecture for Exam Downstream Effects: https://github.com/edx/edx-exams/blob/main/docs/decisions/0004-downstream-effect-events.rst
+.. _How to start using the Event Bus: https://openedx.atlassian.net/wiki/spaces/AC/pages/3508699151/How+to+start+using+the+Event+Bus
+.. _devstack: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/
+.. _Optional Import of Confluent Kafka: https://github.com/openedx/event-bus-kafka/blob/main/docs/decisions/0005-optional-import-of-confluent-kafka.rst.
+.. _manual testing: https://github.com/openedx/event-bus-kafka/blob/main/docs/how_tos/manual_testing.rst
+.. _consume_events management command documentation: https://github.com/openedx/openedx-events/blob/7e6e92429485133bf16ae4494da71b5a2ac31b9e/openedx_events/management/commands/consume_events.py
+
 Setting up an exam and proctoring tool
 --------------------------------------
 

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -1,7 +1,9 @@
 version: "2.1"
 services:
   db:
-    image: mysql:5.6
+    # Oracle-packaged version includes a `linux/arm64/v8` version, needed for
+    # machines with Apple Silicon CPUs (Mac M1, M2)
+    image: mysql:8.0.33-oracle
     container_name: edx_exams.db
     environment:
       MYSQL_ROOT_PASSWORD: ""

diff --git a/edx_exams/apps/api/v1/views.py b/edx_exams/apps/api/v1/views.py
@@ -36,6 +36,7 @@
     get_exam_by_id,
     get_provider_by_exam_id,
     is_exam_passed_due,
+    reset_exam_attempt,
     update_attempt_status
 )
 from edx_exams.apps.core.exam_types import get_exam_type
@@ -626,7 +627,7 @@ def delete(self, request, attempt_id):
             error = {'detail': error_msg}
             return Response(status=status.HTTP_403_FORBIDDEN, data=error)
 
-        exam_attempt.delete()
+        reset_exam_attempt(exam_attempt, request.user)
         return Response(status=status.HTTP_204_NO_CONTENT)
 
 

diff --git a/edx_exams/apps/core/api.py b/edx_exams/apps/core/api.py
@@ -17,6 +17,13 @@
     ExamIllegalStatusTransition
 )
 from edx_exams.apps.core.models import Exam, ExamAttempt
+from edx_exams.apps.core.signals.signals import (
+    emit_exam_attempt_errored_event,
+    emit_exam_attempt_rejected_event,
+    emit_exam_attempt_reset_event,
+    emit_exam_attempt_submitted_event,
+    emit_exam_attempt_verified_event
+)
 from edx_exams.apps.core.statuses import ExamAttemptStatus
 
 log = logging.getLogger(__name__)
@@ -93,15 +100,70 @@ def update_attempt_status(attempt_id, to_status):
         attempt_obj.start_time = datetime.now(pytz.UTC)
         attempt_obj.allowed_time_limit_mins = _calculate_allowed_mins(attempt_obj.exam)
 
+    course_key = CourseKey.from_string(attempt_obj.exam.course_id)
+    usage_key = UsageKey.from_string(attempt_obj.exam.content_id)
+
     if to_status == ExamAttemptStatus.submitted:
         attempt_obj.end_time = datetime.now(pytz.UTC)
 
+        emit_exam_attempt_submitted_event(
+            attempt_obj.user,
+            course_key,
+            usage_key,
+            attempt_obj.exam.exam_type
+        )
+
+    if to_status == ExamAttemptStatus.verified:
+        emit_exam_attempt_verified_event(
+            attempt_obj.user,
+            course_key,
+            usage_key,
+            attempt_obj.exam.exam_type
+        )
+
+    if to_status == ExamAttemptStatus.rejected:
+        emit_exam_attempt_rejected_event(
+            attempt_obj.user,
+            course_key,
+            usage_key,
+            attempt_obj.exam.exam_type
+        )
+
+    if to_status == ExamAttemptStatus.error:
+        emit_exam_attempt_errored_event(
+            attempt_obj.user,
+            course_key,
+            usage_key,
+            attempt_obj.exam.exam_type
+        )
+
     attempt_obj.status = to_status
     attempt_obj.save()
 
     return attempt_id
 
 
+def reset_exam_attempt(attempt, requesting_user):
+    """
+    Reset an exam attempt
+    """
+    course_key = CourseKey.from_string(attempt.exam.course_id)
+    usage_key = UsageKey.from_string(attempt.exam.content_id)
+
+    log.info(
+        f'Resetting exam attempt for user_id={attempt.user.id} in exam={attempt.exam.id} '
+    )
+
+    attempt.delete()
+    emit_exam_attempt_reset_event(
+        attempt.user,
+        course_key,
+        usage_key,
+        attempt.exam.exam_type,
+        requesting_user
+    )
+
+
 def _allow_status_transition(attempt_obj, to_status):
     """
     Helper method to assert that a given status transition is allowed

diff --git a/edx_exams/apps/core/apps.py b/edx_exams/apps/core/apps.py
@@ -0,0 +1,18 @@
+"""
+Core Application Configuration
+"""
+
+from django.apps import AppConfig
+
+
+class CoreConfig(AppConfig):
+    """
+    Application configuration for core application.
+    """
+    name = 'edx_exams.apps.core'
+
+    def ready(self):
+        """
+        Connect handlers to signals.
+        """
+        from .signals import handlers  # pylint: disable=unused-import,import-outside-toplevel
diff --git a/edx_exams/apps/core/migrations/0019_alter_user_full_name.py b/edx_exams/apps/core/migrations/0019_alter_user_full_name.py
@@ -0,0 +1,18 @@
+# Generated by Django 3.2.21 on 2023-10-02 20:21
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0018_staff_roles'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='user',
+            name='full_name',
+            field=models.CharField(blank=True, default='', max_length=255, verbose_name='Full Name'),
+        ),
+    ]
diff --git a/edx_exams/apps/core/models.py b/edx_exams/apps/core/models.py
@@ -24,7 +24,8 @@ class User(AbstractUser):
     .. pii_retirement: local_api
 
     """
-    full_name = models.CharField(_('Full Name'), max_length=255, blank=True, null=True)
+    # The default empty string was added to change full_name from nullable to non-nullable.
+    full_name = models.CharField(_('Full Name'), max_length=255, blank=True, default='')
 
     lms_user_id = models.IntegerField(null=True, db_index=True)
 

diff --git a/edx_exams/apps/core/signals/__init__.py b/edx_exams/apps/core/signals/__init__.py
diff --git a/edx_exams/apps/core/signals/handlers.py b/edx_exams/apps/core/signals/handlers.py
@@ -0,0 +1,85 @@
+"""
+Signal handlers for the edx-exams application.
+"""
+from django.conf import settings
+from django.dispatch import receiver
+from openedx_events.event_bus import get_producer
+from openedx_events.learning.signals import (
+    EXAM_ATTEMPT_ERRORED,
+    EXAM_ATTEMPT_REJECTED,
+    EXAM_ATTEMPT_RESET,
+    EXAM_ATTEMPT_SUBMITTED,
+    EXAM_ATTEMPT_VERIFIED
+)
+
+topic_name = getattr(settings, 'EXAM_ATTEMPT_EVENTS_KAFKA_TOPIC_NAME', '')
+
+
+@receiver(EXAM_ATTEMPT_SUBMITTED)
+def listen_for_exam_attempt_submitted(sender, signal, **kwargs):  # pylint: disable=unused-argument
+    """
+    Publish EXAM_ATTEMPT_SUBMITTED signals onto the event bus.
+    """
+    get_producer().send(
+        signal=EXAM_ATTEMPT_SUBMITTED,
+        topic=topic_name,
+        event_key_field='exam_attempt.course_key',
+        event_data={'exam_attempt': kwargs['exam_attempt']},
+        event_metadata=kwargs['metadata'],
+    )
+
+
+@receiver(EXAM_ATTEMPT_VERIFIED)
+def listen_for_exam_attempt_verified(sender, signal, **kwargs):  # pylint: disable=unused-argument
+    """
+    Publish EXAM_ATTEMPT_VERIFIED signal onto the event bus
+    """
+    get_producer().send(
+        signal=EXAM_ATTEMPT_VERIFIED,
+        topic=topic_name,
+        event_key_field='exam_attempt.course_key',
+        event_data={'exam_attempt': kwargs['exam_attempt']},
+        event_metadata=kwargs['metadata'],
+    )
+
+
+@receiver(EXAM_ATTEMPT_REJECTED)
+def listen_for_exam_attempt_rejected(sender, signal, **kwargs):  # pylint: disable=unused-argument
+    """
+    Publish EXAM_ATTEMPT_REJECTED signal onto the event bus
+    """
+    get_producer().send(
+        signal=EXAM_ATTEMPT_REJECTED,
+        topic=topic_name,
+        event_key_field='exam_attempt.course_key',
+        event_data={'exam_attempt': kwargs['exam_attempt']},
+        event_metadata=kwargs['metadata'],
+    )
+
+
+@receiver(EXAM_ATTEMPT_ERRORED)
+def listen_for_exam_attempt_errored(sender, signal, **kwargs):  # pylint: disable=unused-argument
+    """
+    Publish EXAM_ATTEMPT_ERRORED signal onto the event bus
+    """
+    get_producer().send(
+        signal=EXAM_ATTEMPT_ERRORED,
+        topic=topic_name,
+        event_key_field='exam_attempt.course_key',
+        event_data={'exam_attempt': kwargs['exam_attempt']},
+        event_metadata=kwargs['metadata'],
+    )
+
+
+@receiver(EXAM_ATTEMPT_RESET)
+def listen_for_exam_attempt_reset(sender, signal, **kwargs):  # pylint: disable=unused-argument
+    """
+    Publish EXAM_ATTEMPT_RESET signal onto the event bus
+    """
+    get_producer().send(
+        signal=EXAM_ATTEMPT_RESET,
+        topic=topic_name,
+        event_key_field='exam_attempt.course_key',
+        event_data={'exam_attempt': kwargs['exam_attempt']},
+        event_metadata=kwargs['metadata'],
+    )