Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add sound and browser notifications for agent state changes #6530

Draft
wants to merge 59 commits into
base: main
Choose a base branch
from

Conversation

xingyaoww
Copy link
Collaborator

@xingyaoww xingyaoww commented Jan 29, 2025

Add notification features to improve user experience when the agent needs attention:

  • Sound notifications when agent needs input (with toggle)
  • Browser tab title flashing when window is not focused
  • Tooltips for all trajectory buttons
  • Proper SSR handling and i18n support

Sound button:

image

Browser tab notification:

image

The notifications trigger when agent state changes to:

  • Awaiting user input
  • Awaiting confirmation
  • Task finished

To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:5f34e20-nikolaik   --name openhands-app-5f34e20   docker.all-hands.dev/all-hands-ai/openhands:5f34e20

openhands-agent and others added 30 commits January 29, 2025 22:18
- Reverted vite.config.ts to main branch version
- Reverted package.json and package-lock.json to main branch versions
- Reverted Dockerfile to main branch version to maintain consistent Node.js version
- Added @types/react-router-dom package
- Created custom type declaration for use-sound
- Fixed linting issues
- Installed specific version of use-sound (4.0.1)
- Added howler as a dependency
- Fixed build issues with ESM modules
- Removed frontend/Makefile (not needed)
- Removed frontend/src/main.tsx (using index.tsx instead)
- Reverted changes to frontend/index.html
- Removed duplicate notification.wav from root
- Reverted changes to feedback-actions.test.tsx
- Will add sound toggle tests separately if needed
- Removed separate AgentStateNotifier component
- Integrated notification logic into existing AgentStatusBar component
- Simplified the notification implementation
- Removed favicon modification feature
- Simplified notification permission handling
- Made code more concise while maintaining core functionality
- Removed AgentStateNotifier import and usage since its functionality
  has been integrated into AgentStatusBar
- Added tooltip to sound toggle button
- Fixed sound toggle state update by invalidating settings query
- Added translations for sound toggle tooltips
- Added new TooltipWrapper component for general tooltip needs
- Updated TrajectoryActionButton to use TooltipWrapper
- Fixed build errors
- Removed use-sound package to fix ESM/CommonJS compatibility issues
- Using howler directly for sound playback
- Removed use-sound type declarations
- Added useCallback for better performance
- Added @types/howler package to fix TypeScript errors
- Build now succeeds without ESM/CommonJS issues
- Removed howler and its types (overkill for simple notification)
- Using native HTML5 Audio API instead
- Added error handling for autoplay restrictions
- Moved Audio initialization to useRef to handle SSR
- Added window existence checks
- Improved code organization
- Fixed useRef type definition for HTMLAudioElement
- Added proper undefined initialization
- Added enable_sound_notifications to settings mutation
- Added error handling with toast notification
- Fixed settings update flow
- Added explicit return type for notify function
- Removed console.error statement
- Added explicit undefined returns
- Removed unnecessary TooltipWrapper component
- Use NextUI Tooltip component directly in TrajectoryActionButton
- Added playSound option to notification hook
- Only play sound when explicitly requested
- Fixed unintended sound on settings toggle
- Added browser tab title flashing for notifications
- Added favicon update for notifications
- Auto-clear notifications when window regains focus
- Only show tab notifications when window is not focused
- Added browser environment checks
- Safely initialize title variable
- Added SSR guards in event listeners
- Fixed document/window access during SSR
- Added tooltips for thumbs up, thumbs down, and export buttons
- Added translations for all tooltips in multiple languages
- Improved button accessibility with descriptive tooltips
@xingyaoww xingyaoww changed the title Add Sound and Browser Notifications for Agent State Changes feat: add sound and browser notifications for agent state changes Feb 2, 2025
openhands-agent and others added 8 commits February 2, 2025 06:49
- Updated tooltip translation keys to follow enum format
- Fixed i18n declaration file
- Fixed linting issues in all files
- Restored original gap in agent-control-bar
- Restored original SettingsContext format
- Improved settings update flow to avoid flicker
- Added optimistic updates for settings changes
- Adjusted query settings to reduce errors
- Added retry logic for failed requests
- Prevent sound from playing when enabling notifications
- Add visual feedback (toast) when toggling sound
- Add translations for sound toggle messages
- Only play sounds for agent state changes
- Fixed i18n key format to follow enum pattern
- Removed unnecessary toast message on sound toggle
- Keep error toast for failed settings updates
- Added stricter conditions for playing notification sounds
- Explicitly check for BUTTON$ prefix to avoid settings sounds
- Fixed translation key usage in notifications
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants