Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT: Unified Notifications #290

Draft
wants to merge 66 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
0d7f44a
added logs component to layout
jagadeeswaran-zipstack Apr 25, 2024
9b94cc8
sonar fix
jagadeeswaran-zipstack Apr 25, 2024
4e36241
added border and fixed layout
jagadeeswaran-zipstack Apr 28, 2024
ef0d76a
Merge branch 'main' into feat/unified-notifications
jagadeeswaran-zipstack Apr 28, 2024
8d01c7a
added close and close all button
jagadeeswaran-zipstack Apr 30, 2024
a8e7e05
Merge branch 'main' into feat/unified-notifications
tahierhussain May 1, 2024
b92adf0
Merge branch 'main' of github.com:Zipstack/unstract into feat/unified…
tahierhussain May 9, 2024
a80b716
Backend related changes for unified notifications
tahierhussain May 13, 2024
2bde1d4
Changes in pubsub helper
tahierhussain May 13, 2024
bef8109
UI related changes in unified notifications
tahierhussain May 13, 2024
dce29af
Merge branch 'main' of github.com:Zipstack/unstract into feat/unified…
tahierhussain May 13, 2024
512222d
Minor fixes
tahierhussain May 13, 2024
c285e41
Merge branch 'main' into feat/unified-notifications
tahierhussain May 13, 2024
753b4df
Merge branch 'main' into feat/unified-notifications
tahierhussain May 14, 2024
8a9f8a4
Display log messages from the API response
tahierhussain May 21, 2024
7065e28
Merge branch 'feat/unified-notifications' of github.com:Zipstack/unst…
tahierhussain May 21, 2024
76d5ade
Merge branch 'main' into feat/unified-notifications
jagadeeswaran-zipstack May 29, 2024
0526d35
store notifications to redis logs
jagadeeswaran-zipstack May 29, 2024
4e3d00f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 29, 2024
4dd9b40
Merge branch 'main' into feat/unified-notifications
tahierhussain May 31, 2024
b30cc6d
updated sample.env
jagadeeswaran-zipstack May 31, 2024
71c4de7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 31, 2024
f4ae7fc
Merge branch 'main' of github.com:Zipstack/unstract into feat/unified…
tahierhussain Jun 3, 2024
0d91070
Removed exceptions as it is handled by default
tahierhussain Jun 3, 2024
b091bd7
changed to django redis
jagadeeswaran-zipstack Jun 3, 2024
c6e9f5d
Merge branch 'main' into feat/unified-notifications
jagadeeswaran-zipstack Jun 3, 2024
23318c2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 3, 2024
b3eaf66
build fix
jagadeeswaran-zipstack Jun 3, 2024
6806d51
remove logs on logout
jagadeeswaran-zipstack Jun 5, 2024
6b1949f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 5, 2024
c25f2ea
pre-commit fix
jagadeeswaran-zipstack Jun 5, 2024
6e188ef
Merge branch 'feat/unified-notifications' of github.com:Zipstack/unst…
jagadeeswaran-zipstack Jun 5, 2024
1e02027
Merge branch 'main' into feat/unified-notifications
tahierhussain Jun 6, 2024
91af677
reused CacheService
jagadeeswaran-zipstack Jun 10, 2024
b616fa0
Merge branch 'feat/unified-notifications' of github.com:Zipstack/unst…
jagadeeswaran-zipstack Jun 10, 2024
fb74901
Merge branch 'main' into feat/unified-notifications
jagadeeswaran-zipstack Jun 10, 2024
c451c51
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 10, 2024
879863f
Merge branch 'feat/unified-notifications' of github.com:Zipstack/unst…
jagadeeswaran-zipstack Jun 10, 2024
c643b02
merge fix
jagadeeswaran-zipstack Jun 10, 2024
68516da
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 10, 2024
b8f2a33
fixed return type
jagadeeswaran-zipstack Jun 10, 2024
4054631
Merge branch 'feat/unified-notifications' of github.com:Zipstack/unst…
jagadeeswaran-zipstack Jun 10, 2024
7e9511f
code cleanup
jagadeeswaran-zipstack Jun 20, 2024
c0fcfde
code cleanup
jagadeeswaran-zipstack Jun 20, 2024
2619260
encapsulated redis key creation
jagadeeswaran-zipstack Jun 24, 2024
08b9a9a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 24, 2024
e9badc3
code optimization
jagadeeswaran-zipstack Jun 24, 2024
5f43ca2
Merge branch 'feat/unified-notifications' of github.com:Zipstack/unst…
jagadeeswaran-zipstack Jun 24, 2024
abc8f71
Merge branch 'main' of github.com:Zipstack/unstract into feat/unified…
jagadeeswaran-zipstack Jul 15, 2024
30c7203
Merge branch 'main' of github.com:Zipstack/unstract into feat/unified…
jagadeeswaran-zipstack Jul 15, 2024
e467d8b
Merge branch 'main' of github.com:Zipstack/unstract into feat/unified…
jagadeeswaran-zipstack Jul 19, 2024
e898b18
added logs to redis
jagadeeswaran-zipstack Jul 29, 2024
cbe6116
Merge branch 'main' of github.com:Zipstack/unstract into feat/unified…
tahierhussain Jul 30, 2024
45a9bb1
UI/UX improvements in unified notifications
tahierhussain Jul 30, 2024
8ee5c59
Handled default duration of popup message
tahierhussain Jul 30, 2024
2de8a68
sonar issue fix
jagadeeswaran-zipstack Jul 30, 2024
fc42be6
Merge branch 'main' into feat/unified-notifications
tahierhussain Jul 31, 2024
a1be308
settings changes
jagadeeswaran-zipstack Jul 31, 2024
cfd496b
Merge branch 'feat/unified-notifications' of github.com:Zipstack/unst…
jagadeeswaran-zipstack Jul 31, 2024
808c31e
code refactor
jagadeeswaran-zipstack Jul 31, 2024
fa633cf
moved inline style to class
jagadeeswaran-zipstack Jul 31, 2024
5a2c170
Merge branch 'main' into feat/unified-notifications
tahierhussain Jul 31, 2024
0a0f8f2
Merge branch 'main' into feat/unified-notifications
Deepak-Kesavan Jul 31, 2024
b18261c
moved from django-redis to redis
jagadeeswaran-zipstack Jul 31, 2024
69e11df
Merge branch 'feat/unified-notifications' of github.com:Zipstack/unst…
jagadeeswaran-zipstack Jul 31, 2024
66ca748
Merge branch 'main' into feat/unified-notifications
jagadeeswaran-zipstack Aug 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions backend/account/authentication_controller.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
from django.middleware import csrf
from django.shortcuts import redirect
from django_tenants.utils import tenant_context
from logs_helper.log_service import LogService
from rest_framework import status
from rest_framework.request import Request
from rest_framework.response import Response
Expand Down Expand Up @@ -259,6 +260,8 @@ def make_user_organization_display_name(self, user_name: str) -> str:
return self.auth_service.make_user_organization_display_name(user_name)

def user_logout(self, request: Request) -> Response:
session_id: str = request.COOKIES.get("sessionid")
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved
LogService.remove_logs_on_logout(session_id=session_id)
response = self.auth_service.user_logout(request=request)
organization_id = UserSessionUtils.get_organization_id(request)
user_id = UserSessionUtils.get_user_id(request)
Expand Down
17 changes: 17 additions & 0 deletions backend/logs_helper/log_service.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
from django_redis import get_redis_connection


class LogService:
@staticmethod
def remove_logs_on_logout(session_id):
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved

if session_id:
# Get the Redis connection
r = get_redis_connection("default")
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved

key_pattern = f"logs:{session_id}*"
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved

# Retrieve keys matching the pattern and delete them
keys = r.keys(key_pattern)
athul-rs marked this conversation as resolved.
Show resolved Hide resolved
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved
if keys:
r.delete(*keys)
4 changes: 2 additions & 2 deletions backend/logs_helper/urls.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
from django.urls import path
from rest_framework.urlpatterns import format_suffix_patterns

from .views import LogsHelperView
from .views import LogsHelperViewSet

logs_helper_get = LogsHelperView.as_view({"get": "get_logs", "post": "store_log"})
logs_helper_get = LogsHelperViewSet.as_view({"get": "get_logs", "post": "store_log"})

urlpatterns = format_suffix_patterns(
[
Expand Down
109 changes: 41 additions & 68 deletions backend/logs_helper/views.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,10 @@
import os
from datetime import datetime, timezone

import redis
from django_redis import get_redis_connection
from logs_helper.constants import LogsHelperKeys
from logs_helper.exceptions import InvalidValueError, MissingFieldsKeyError
from rest_framework import status, viewsets
from rest_framework.decorators import action
from rest_framework.exceptions import APIException
from rest_framework.response import Response
from utils.local_context import StateStore

Expand All @@ -17,77 +15,52 @@
logger = logging.getLogger(__name__)


class LogsHelperView(viewsets.ModelViewSet):
class LogsHelperViewSet(viewsets.ModelViewSet):
"""Viewset to handle all Tool Studio prompt related API logics."""

r = redis.Redis(
host=os.environ.get("REDIS_HOST", "http://localhost"),
port=os.environ.get("REDIS_PORT", "6379"),
username=os.environ.get("REDIS_USER", ""),
password=os.environ.get("REDIS_PASSWORD", ""),
)
r = get_redis_connection("default")
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved

@action(detail=False, methods=["get"])
def get_logs(self, request):
try:
# Extract the session ID
session_id: str = StateStore.get(LogsHelperKeys.LOG_EVENTS_ID)

# Construct the Redis key pattern to match keys
# associated with the session ID
redis_key = f"logs:{session_id}*"

# Retrieve keys matching the pattern
keys = self.r.keys(redis_key)

# Retrieve values corresponding to the keys and sort them by timestamp
logs = []
for key in keys:
log_data = self.r.get(key).decode()
log_entry = json.loads(log_data)
logs.append(log_entry)

# Sort logs based on timestamp
sorted_logs = sorted(logs, key=lambda x: x["timestamp"])

return Response({"data": sorted_logs}, status=status.HTTP_200_OK)
except Exception as e:
# Handle other exceptions
error_msg = "An unexpected error occurred while retrieving logs"
logger.error(f"{error_msg}: {e}")
raise APIException(error_msg)
# Extract the session ID
session_id: str = StateStore.get(LogsHelperKeys.LOG_EVENTS_ID)
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved

# Construct the Redis key pattern to match keys
# associated with the session ID
redis_key = f"logs:{session_id}*"

# Retrieve keys matching the pattern
keys = self.r.keys(redis_key)
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved

# Retrieve values corresponding to the keys and sort them by timestamp
logs = []
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved
for key in keys:
log_data = self.r.get(key).decode()
log_entry = json.loads(log_data)
logs.append(log_entry)

# Sort logs based on timestamp
sorted_logs = sorted(logs, key=lambda x: x["timestamp"])

return Response({"data": sorted_logs}, status=status.HTTP_200_OK)
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved

@action(detail=False, methods=["post"])
def store_log(self, request):
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved
"""Store log message in Redis."""
try:
# Extract the session ID
logs_expiry = int(os.environ.get("LOGS_EXPIRATION_TIME_IN_SECOND", 3600))
session_id: str = StateStore.get(LogsHelperKeys.LOG_EVENTS_ID)

serializer = StoreLogMessagesSerializer(data=request.data)
serializer.is_valid(raise_exception=True)

# Extract the log message from the validated data
log: str = serializer.validated_data.get("log")

timestamp = datetime.now(timezone.utc).timestamp()

redis_key = f"logs:{session_id}:{timestamp}"

self.r.setex(redis_key, logs_expiry, log)

return Response({"message": "Successfully stored the message in redis"})
except KeyError as e:
# Handle KeyError
logger.error(f"Log is missing: {e}")
raise MissingFieldsKeyError()
except ValueError as e:
# Handle ValueError
logger.error(f"Invalid value: {e}")
raise InvalidValueError()
except Exception as e:
# Handle other exceptions
error_msg = "An unexpected error occurred while store the log message"
logger.error(f"{error_msg}: {e}")
raise APIException(error_msg)
# Extract the session ID
logs_expiry = int(os.environ.get("LOGS_EXPIRATION_TIME_IN_SECOND", 3600))
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved
session_id: str = StateStore.get(LogsHelperKeys.LOG_EVENTS_ID)
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved

serializer = StoreLogMessagesSerializer(data=request.data)
serializer.is_valid(raise_exception=True)

# Extract the log message from the validated data
log: str = serializer.validated_data.get("log")

timestamp = datetime.now(timezone.utc).timestamp()

redis_key = f"logs:{session_id}:{timestamp}"

self.r.setex(redis_key, logs_expiry, log)
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved

return Response({"message": "Successfully stored the message in redis"})
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ function CustomToolsHelper() {
useEffect(() => {
return () => {
setDefaultCustomTool();
resetTokenUsage();
};
}, []);

Expand Down
2 changes: 1 addition & 1 deletion frontend/src/layouts/page-layout/PageLayout.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ function PageLayout() {
const response = await axios(requestOptions);
return response.data;
} catch (error) {
return;
return { data: [] };
}
};

Expand Down
9 changes: 2 additions & 7 deletions unstract/core/src/unstract/core/pubsub_helper.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,11 @@
from datetime import datetime, timezone
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved
from typing import Any, Optional

import redis
from django_redis import get_redis_connection
jagadeeswaran-zipstack marked this conversation as resolved.
Show resolved Hide resolved


class LogPublisher:
r = redis.Redis(
host=os.environ.get("REDIS_HOST", "http://localhost"),
port=os.environ.get("REDIS_PORT", "6379"),
username=os.environ.get("REDIS_USER", ""),
password=os.environ.get("REDIS_PASSWORD", ""),
)
r = get_redis_connection("default")

@staticmethod
def log_usage(
Expand Down
Loading