[To Main] Feature/DESENG-445: engagement filtering by metadata (#2444)

- **Feature** Engagement filtering - Add filtering by taxon [🎟️DESENG-445](https://apps.itsm.gov.bc.ca/jira/browse/DESENG-445) - Added properties to metadata taxa to allow them to be marked as filterable. - Added a new file `filter_types.py` where additional filters can be added as subquery factory functions. - Added a new endpoint to the API to retrieve filterable taxa. - Modified the search endpoint to allow filtering by metadata. - Added schemas, data validation and unit tests for the new functionality. - Updated the Metadata Management UI to allow taxa to be marked as filterable. - Currently, the only two filter types are `chips_any` and `chips_all`. - `chips_any`: Displays as a series of toggleable buttons ("chips"), uses the `list_match_any` subquery returning engagements with any of the selected values. - `chips_all`: Similar to chips_any; uses the `list_match_all` subquery to get only engagements with ALL of the selected values. - If multiple filterable taxa are selected, all the taxon filters must be met for an engagement to be returned. - Updated the public-facing engagement list to allow filtering by metadata taxa. This makes use of the new API endpoint to retrieve filterable taxa. - Added a new filter "drawer" to the listing page to hold these and any future filter types. - (Out of scope, but related to UX work for this ticket) Fixed a display issue with the public engagements page where engagements would not take up the full width of their grid cell. --------- Co-authored-by: Ratheesh kumar R <[email protected]> Co-authored-by: VineetBala-AOT <[email protected]> Co-authored-by: Baelx <[email protected]>
bcgov · Apr 5, 2024 · bbbaddf · bbbaddf
1 parent 51d101f
commit bbbaddf
Show file tree

Hide file tree

Showing 40 changed files with 1,894 additions and 374 deletions.
diff --git a/CHANGELOG.MD b/CHANGELOG.MD
@@ -1,5 +1,21 @@
 ## April 04, 2024
 
+- **Feature** Engagement filtering - Add filtering by taxon [🎟️DESENG-445](https://apps.itsm.gov.bc.ca/jira/browse/DESENG-445)
+
+  - Added properties to metadata taxa to allow them to be marked as filterable.
+  - Added a new file `filter_types.py` where additional filters can be added as subquery factory functions.
+  - Added a new endpoint to the API to retrieve filterable taxa.
+  - Modified the search endpoint to allow filtering by metadata.
+  - Added schemas, data validation and unit tests for the new functionality.
+  - Updated the Metadata Management UI to allow taxa to be marked as filterable.
+    - Currently, the only two filter types are `chips_any` and `chips_all`.
+    - `chips_any`: Displays as a series of toggleable buttons ("chips"), uses the `list_match_any` subquery returning engagements with any of the selected values.
+    - `chips_all`: Similar to chips_any; uses the `list_match_all` subquery to get only engagements with ALL of the selected values.
+  - If multiple filterable taxa are selected, all the taxon filters must be met for an engagement to be returned.
+  - Updated the public-facing engagement list to allow filtering by metadata taxa. This makes use of the new API endpoint to retrieve filterable taxa.
+  - Added a new filter "drawer" to the listing page to hold these and any future filter types.
+  - (Out of scope, but related to UX work for this ticket) Fixed a display issue with the public engagements page where engagements would not take up the full width of their grid cell.
+
 - **Task**: Keycloak Unit Tests for New CSS API Integration [DESENG-508](https://apps.itsm.gov.bc.ca/jira/browse/DESENG-508)
   - Updated Keycloak Unit Tests for New CSS API Integration.
 
@@ -25,7 +41,7 @@
 
 - **Bug Fix**: Various bugs in the survey tab [DESENG-520](https://apps.itsm.gov.bc.ca/jira/browse/DESENG-523)
   - Update survey components and services to fix the bugs.
-    -  Removed is_template and is_hidden values from autosaving to avoid the debounce function consuming it older values
+    - Removed is_template and is_hidden values from autosaving to avoid the debounce function consuming it older values
 - **Task**: Add a custom dynamic page to engagement [DESENG-501](https://apps.itsm.gov.bc.ca/jira/browse/DESENG-501)
   - Integrated new summary and custom content tabs into the engagement form, providing users with structured sections for managing different types of content, enhancing overall organization and user experience.
   - Implemented functionality to allow users to add custom tabs with a variety of provided icons, along with options to edit or delete tabs as needed, granting users greater flexibility and control over their engagement content layout.
@@ -44,7 +60,6 @@
   - Added poll results to results tab.
   - Added poll results API.
   - Added Unit tests.
-
 - **Task**: Change static english text to be able to support string translations [DESENG-467](https://apps.itsm.gov.bc.ca/jira/browse/DESENG-467)
   - Implemented a language selector in the public header.
   - Incorporated logic to dynamically adjust the unauthenticated route based on the selected language and load the appropriate translation file.

diff --git a/met-api/migrations/versions/f8bc8ce202f3_add_metadata_filters.py b/met-api/migrations/versions/f8bc8ce202f3_add_metadata_filters.py
@@ -0,0 +1,32 @@
+"""Add metadata filters to the engagement metadata taxa table.
+
+Revision ID: f8bc8ce202f3
+Revises: 734f160dd120
+Create Date: 2024-03-21 13:02:12.680363
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = 'f8bc8ce202f3'
+down_revision = '734f160dd120'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('engagement_metadata_taxa', sa.Column(
+        'filter_type', sa.String(length=64), nullable=True))
+    op.add_column('engagement_metadata_taxa', sa.Column(
+        'include_freeform', sa.Boolean(), nullable=False, server_default='false'))
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('engagement_metadata_taxa', 'include_freeform')
+    op.drop_column('engagement_metadata_taxa', 'filter_type')
+    # ### end Alembic commands ###
diff --git a/met-api/src/met_api/models/engagement.py b/met-api/src/met_api/models/engagement.py
@@ -15,15 +15,18 @@
 
 from met_api.constants.engagement_status import EngagementDisplayStatus, Status
 from met_api.constants.user import SYSTEM_USER
+from met_api.models.engagement_scope_options import EngagementScopeOptions
 from met_api.models.membership import Membership as MembershipModel
-from met_api.models.staff_user import StaffUser
 from met_api.models.pagination_options import PaginationOptions
-from met_api.models.engagement_scope_options import EngagementScopeOptions
+from met_api.models.staff_user import StaffUser
 from met_api.schemas.engagement import EngagementSchema
 from met_api.utils.datetime import local_datetime
 from met_api.utils.enums import MembershipStatus
+from met_api.utils.filter_types import filter_map
+
 from .base_model import BaseModel
 from .db import db
+from .engagement_metadata import EngagementMetadata as EngagementMetadataModel
 from .engagement_status import EngagementStatus
 
 
@@ -37,14 +40,18 @@ class Engagement(BaseModel):
     rich_description = db.Column(JSON, unique=False, nullable=False)
     start_date = db.Column(db.DateTime)
     end_date = db.Column(db.DateTime)
-    status_id = db.Column(db.Integer, ForeignKey('engagement_status.id', ondelete='CASCADE'))
+    status_id = db.Column(db.Integer, ForeignKey(
+        'engagement_status.id', ondelete='CASCADE'))
     status = db.relationship('EngagementStatus', backref='engagement')
     published_date = db.Column(db.DateTime, nullable=True)
     scheduled_date = db.Column(db.DateTime, nullable=True)
     banner_filename = db.Column(db.String(), unique=False, nullable=True)
-    surveys = db.relationship('Survey', backref='engagement', cascade='all, delete')
-    status_block = db.relationship('EngagementStatusBlock', backref='engagement')
-    tenant_id = db.Column(db.Integer, db.ForeignKey('tenant.id'), nullable=True)
+    surveys = db.relationship(
+        'Survey', backref='engagement', cascade='all, delete')
+    status_block = db.relationship(
+        'EngagementStatusBlock', backref='engagement')
+    tenant_id = db.Column(
+        db.Integer, db.ForeignKey('tenant.id'), nullable=True)
     tenant = db.relationship('Tenant', backref='engagements')
     is_internal = db.Column(db.Boolean, nullable=False)
     consent_message = db.Column(JSON, unique=False, nullable=True)
@@ -71,16 +78,16 @@ def get_engagements_paginated(
 
             query = cls._filter_by_engagement_status(query, search_options)
 
-            # TODO: Uncomment for metadata changes coming soon.
-            # query = cls._filter_by_project_metadata(query, search_options)
+            query = cls._filter_by_metadata(query, search_options)
 
         query = cls._filter_by_internal(query, search_options)
 
         if scope_options.restricted:
             if scope_options.include_assigned:
                 # the engagement status ids that should not be filtered out
                 exception_status_ids = scope_options.engagement_status_ids
-                query = cls._filter_by_assigned_engagements(query, external_user_id, exception_status_ids)
+                query = cls._filter_by_assigned_engagements(
+                    query, external_user_id, exception_status_ids)
             else:
                 # the engagement status ids of the engagements that should fetched
                 statuses = scope_options.engagement_status_ids
@@ -95,7 +102,8 @@ def get_engagements_paginated(
             items = query.all()
             return items, len(items)
 
-        page = query.paginate(page=pagination_options.page, per_page=pagination_options.size)
+        page = query.paginate(page=pagination_options.page,
+                              per_page=pagination_options.size)
         return page.items, page.total
 
     @classmethod
@@ -116,13 +124,16 @@ def update_engagement(cls, engagement: EngagementSchema) -> Engagement:
             status_id=engagement.get('status_id', None),
             # to fix the bug with UI not passing published date always.
             # Defaulting to existing
-            published_date=engagement.get('published_date', record.published_date),
-            scheduled_date=engagement.get('scheduled_date', record.scheduled_date),
+            published_date=engagement.get(
+                'published_date', record.published_date),
+            scheduled_date=engagement.get(
+                'scheduled_date', record.scheduled_date),
             updated_date=datetime.utcnow(),
             updated_by=engagement.get('updated_by', None),
             banner_filename=engagement.get('banner_filename', None),
             is_internal=engagement.get('is_internal', record.is_internal),
-            consent_message=engagement.get('consent_message', record.consent_message),
+            consent_message=engagement.get(
+                'consent_message', record.consent_message),
         )
         query.update(update_fields)
         db.session.commit()
@@ -198,19 +209,20 @@ def _get_sort_order(pagination_options):
 
     @staticmethod
     def _filter_by_engagement_status(query, search_options):
-        statuses = [int(status) for status in search_options.get('engagement_status', [])]
+        statuses = [int(status)
+                    for status in search_options.get('engagement_status', [])]
         if not statuses:
             return query
 
-        status_filter = []
-        if EngagementDisplayStatus.Draft.value in statuses:
-            status_filter.append(Engagement.status_id == Status.Draft.value)
-        if EngagementDisplayStatus.Published.value in statuses:
-            status_filter.append(Engagement.status_id == Status.Published.value)
-        if EngagementDisplayStatus.Closed.value in statuses:
-            status_filter.append(Engagement.status_id == Status.Closed.value)
-        if EngagementDisplayStatus.Scheduled.value in statuses:
-            status_filter.append(Engagement.status_id == Status.Scheduled.value)
+        allowed_statuses = [
+            Status.Draft.value,
+            Status.Published.value,
+            Status.Closed.value,
+            Status.Scheduled.value
+        ]
+        status_filter = [Engagement.status_id.in_(
+            [status for status in statuses if status in allowed_statuses])]
+
         if EngagementDisplayStatus.Upcoming.value in statuses:
             status_filter.append(
                 and_(
@@ -226,16 +238,19 @@ def _filter_by_engagement_status(query, search_options):
                 )
             )
         if EngagementDisplayStatus.Unpublished.value in statuses:
-            status_filter.append(Engagement.status_id == Status.Unpublished.value)
+            status_filter.append(Engagement.status_id ==
+                                 Status.Unpublished.value)
         query = query.filter(or_(*status_filter))
         return query
 
     @classmethod
     def _filter_by_published_date(cls, query, search_options):
         if published_from_date := search_options.get('published_from_date'):
-            query = query.filter(Engagement.published_date >= published_from_date)
+            query = query.filter(
+                Engagement.published_date >= published_from_date)
         if published_to_date := search_options.get('published_to_date'):
-            query = query.filter(Engagement.published_date <= published_to_date)
+            query = query.filter(
+                Engagement.published_date <= published_to_date)
         return query
 
     @staticmethod
@@ -249,7 +264,8 @@ def _filter_by_created_date(query, search_options):
     @staticmethod
     def _filter_by_search_text(query, search_options):
         if search_text := search_options.get('search_text'):
-            query = query.filter(Engagement.name.ilike('%' + search_text + '%'))
+            query = query.filter(
+                Engagement.name.ilike('%' + search_text + '%'))
         return query
 
     @staticmethod
@@ -259,11 +275,36 @@ def _filter_by_internal(query, search_options):
                 query = query.filter(Engagement.is_internal.is_(False))
         return query
 
-    # TODO: Populate or remove this method dependent on changes resulting from adding the new Engagement metadata
-    # @staticmethod
-    # def _filter_by_project_metadata(query, search_options):
-    #     query = query.outerjoin(EngagementMetadataModel, EngagementMetadataModel.engagement_id == Engagement.id)
-    #     return query
+    @staticmethod
+    def _filter_by_metadata(query, search_options):
+        """
+        Filter the engagements based on metadata criteria.
+
+        Ensures that each engagement matches all of the provided criteria.
+        """
+        if 'metadata' not in search_options:
+            return query
+
+        for criterion in search_options['metadata']:
+            taxon_id = criterion.get('taxon_id')
+            values = criterion.get('values')
+            # pick the type of filtering to apply
+            filter_type = filter_map.get(criterion.get('filter_type'))
+            if any([taxon_id is None, values is None, filter_type is None]):
+                continue  # skip criterion if any of the required fields are missing
+
+            taxon_query = query.session.query(
+                EngagementMetadataModel.engagement_id
+            ).filter(
+                # Filter the metadata entries to only include those that match the current taxon
+                EngagementMetadataModel.taxon_id == taxon_id
+            )
+            # Use the filter function to create a subquery that filters the engagements
+            filter_subquery = filter_type(taxon_query, values)
+            # Filter the main query to include only engagements found in the subquery
+            query = query.filter(Engagement.id.in_(filter_subquery))
+
+        return query
 
     @staticmethod
     def _filter_by_assigned_engagements(query, external_user_id: int, exception_status_ids: Optional[list[int]] = None):

diff --git a/met-api/src/met_api/models/engagement_metadata.py b/met-api/src/met_api/models/engagement_metadata.py
@@ -89,6 +89,13 @@ def has_value(cls, value: str) -> bool:
         return value in cls._value2member_map_
 
 
+class MetadataTaxonFilterType(str, enum.Enum):
+    """The filter types that can be applied to a metadata property."""
+
+    CHIPS_ALL = 'chips_all'
+    CHIPS_ANY = 'chips_any'
+
+
 class MetadataTaxon(BaseModel):
     """
     A taxon to group metadata by.
@@ -110,6 +117,9 @@ class MetadataTaxon(BaseModel):
     data_type = db.Column(db.String(64), nullable=True, default='text')
     one_per_engagement = db.Column(db.Boolean)
     position = db.Column(db.Integer, nullable=False, index=True)
+    filter_type = db.Column(db.String(64), nullable=True)
+    # Whether to include freeform values from engagements in the user-facing filter options
+    include_freeform = db.Column(db.Boolean, nullable=False, default=False)
 
     def __init__(self, **kwargs) -> None:
         """Initialize a new instance of the MetadataTaxon class."""

diff --git a/met-api/src/met_api/resources/engagement.py b/met-api/src/met_api/resources/engagement.py
@@ -15,6 +15,7 @@
 
 from http import HTTPStatus
 
+import json
 from flask import request
 from flask_cors import cross_origin
 from flask_restx import Namespace, Resource
@@ -30,7 +31,8 @@
 from met_api.utils.token_info import TokenInfo
 from met_api.utils.util import allowedorigins, cors_preflight
 
-API = Namespace('engagements', description='Endpoints for Engagements Management')
+API = Namespace(
+    'engagements', description='Endpoints for Engagements Management')
 """Custom exception messages
 """
 
@@ -83,16 +85,21 @@ def get():
             if external_user_id is None:
                 exclude_internal = True
 
+            metadata = args.getlist('metadata[]')
+            if metadata:
+                metadata = [json.loads(m) for m in metadata]
+
             search_options = {
                 'search_text': args.get('search_text', '', type=str),
                 'engagement_status': args.getlist('engagement_status[]'),
                 'created_from_date': args.get('created_from_date', None, type=str),
                 'created_to_date': args.get('created_to_date', None, type=str),
                 'published_from_date': args.get('published_from_date', None, type=str),
                 'published_to_date': args.get('published_to_date', None, type=str),
+                'metadata': metadata,
                 'exclude_internal': exclude_internal,
-                # the membership changing pages sometimes needs only engagement where user can add a member.
-                # pass this has_team_access to restrict searches only within engagements he has access on.
+                # the membership changing pages sometimes need only engagements where users can add a member.
+                # pass this has_team_access to restrict searches only within engagements they have access on.
                 'has_team_access': args.get(
                     'has_team_access',
                     default=False,