Merge pull request #40 from nicobrenner/nico/too-many-changes

First test and some cleanup

README.md

@@ -27,21 +27,16 @@ Note: If you want to add another source of job listings, [go to this issue](http
 ## Updates
 
 * Building in public:
-    * Was checking out the super cool [ancv](https://github.com/alexpovel/ancv), a tool for building a really cool ascii version of your resume on the terminal! 🤗 (love the joke with the Venn diagram) I wanted to check it out and ended up writing a prompt for their `README` to help with getting some json data out of a resume (which is very similar to what Command Jobs does under the hood as well). Here's the video:
+    * Just wrote the first test! 😅 And it's in no small part thanks to Agentic's [Glide](https://glide.agenticlabs.com/task/IqHd0RV), which they recently launched ([see announcement here](https://news.ycombinator.com/item?id=39682183)). I was about to switch from ncurses to [python-prompt-toolkit](https://github.com/prompt-toolkit/python-prompt-toolkit), and failing that from python to Go, so I could build Command Jobs using [Bubble Tea](https://github.com/charmbracelet/bubbletea) 🤩😍🤤
 
-        * [![Trying ancv](https://cdn.loom.com/sessions/thumbnails/08fe5707eb0e46349ec55be6b2b446aa-with-play.gif)](https://www.loom.com/share/08fe5707eb0e46349ec55be6b2b446aa)
+        * [![First test with Glide](https://cdn.loom.com/sessions/thumbnails/afd0733ac8dd477cbeea63c8ea6cb363-with-play.gif)](https://www.loom.com/share/afd0733ac8dd477cbeea63c8ea6cb363)
 
+    * Check out the amazing [ancv](https://github.com/alexpovel/ancv), a tool for building a really cool ascii version of your resume on the terminal! 🤗 (love the joke with the Venn diagram). Will need to integrate it as a library with Command Jobs
 
-    * Tried out [ShellGPT](https://github.com/mattvr/ShellGPT) and made a small PR to highlight its chat interface in the `README`. It's a pretty cool tool to use GPT from the terminal
+    * Tried out [ShellGPT](https://github.com/mattvr/ShellGPT) and made a small PR to highlight its chat interface in the `README`. It's a pretty cool tool to use GPT from the terminal. Next I want to try coding a bit with [aider](https://github.com/paul-gauthier/aider)
 
         * [![ShellGPT](https://cdn.loom.com/sessions/thumbnails/7f415a53cb404cb0a059a9a065addce8-with-play.gif)](https://www.loom.com/share/7f415a53cb404cb0a059a9a065addce8)
 
-
-    * Finally was able to close out [#12](https://github.com/nicobrenner/commandjobs/issues/12) follow along as I resolve it in the video below
-
-        * [![Fixing #12](https://cdn.loom.com/sessions/thumbnails/9ff310f1a7534b2793b3ed366e9859ac-with-play.gif)](https://www.loom.com/share/9ff310f1a7534b2793b3ed366e9859ac)
-
-
     * Decided to try to build this project as openly as possible, in that spirit, I just recorded a coding session in which I go through the process of trying to resolve a bug ([issue #12](https://github.com/nicobrenner/commandjobs/issues/12)), and finding 3 other bugs instead!
 
         If you are just getting started with coding, it's also a pretty good overview of a basic software project management. In the video I show the whole workflow of not only writing code, but also managing an environment, dealing with errors, documenting the process in Github, managing git and branches, commiting, pushing and merging code, updating documentation (like now), and sharing/promoting

src/database_manager.py

@@ -42,6 +42,15 @@ def fetch_job_listings(self, listings_per_batch):
         """
         self.cursor.execute(query)
         return self.cursor.fetchall()
+
+    def fetch_processed_listings_count(self):
+        query = "SELECT COUNT(id) FROM gpt_interactions"
+        self.cursor.execute(query)
+        result = self.cursor.fetchone()  # Fetch the first row of the result set
+        if result:
+            return result[0]  # Return the first element of the tuple, which is the count
+        else:
+            return 0  # Return 0 if no rows are found, for safety
 
     def save_gpt_interaction(self, job_id, prompt, answer):
         self.cursor.execute("INSERT INTO gpt_interactions (job_id, prompt, answer) VALUES (?, ?, ?)", (job_id, prompt, answer))

src/display_matching_table.py

@@ -11,10 +11,6 @@ def __init__(self, stdscr, db_path):
         self.highlighted_row_index = 0
         self.current_page = 1
         self.total_pages = 0
-        curses.init_pair(3, curses.COLOR_WHITE, curses.COLOR_BLUE)  # Highlight color
-        curses.init_pair(4, curses.COLOR_BLACK, curses.COLOR_WHITE)  # Highlight headers color
-        curses.init_pair(5, curses.COLOR_WHITE, curses.COLOR_MAGENTA)  # Highlight headers color
-        curses.init_pair(6, curses.COLOR_RED, curses.COLOR_BLACK)  # Highlight headers color
         self.rows_per_page = 3
         logging.basicConfig(filename='matching_table_display.log', level=logging.DEBUG)
 

src/hn_scraping.py → src/hn_scraper.py

@@ -9,7 +9,7 @@ class ScrapingInterrupt(Exception):
     pass
 
 class HNScraper:
-    def __init__(self, db_path='/repo/job_listings.db'):
+    def __init__(self, db_path='job_listings.db'):
         self.db_path = db_path
         # Define the base URL for Ask HN: Who's hiring
         self.base_url = 'https://news.ycombinator.com/item?id=39562986&p=1'

src/menu.py

@@ -1,7 +1,7 @@
 import curses
 import textwrap
 import os
-from hn_scraping import HNScraper
+from hn_scraper import HNScraper
 from display_table import draw_table
 from database_manager import DatabaseManager
 from display_matching_table import MatchingTableDisplay
@@ -42,6 +42,8 @@ def __init__(self, stdscr, logger):
 
         self.scraping_done_event = threading.Event()  # Event to signal scraping completion
         self.logger = logger
+        self.stdscr = stdscr
+        self.setup_ncurses()
         self.db_path = os.getenv('DB_PATH')
         if self.db_path is None:
             # I don't like raising an exception here, but haven't been able to
@@ -52,9 +54,9 @@ def __init__(self, stdscr, logger):
         self.db_manager = DatabaseManager(self.db_path)  # Specify the path
         self.gpt_processor = GPTProcessor(self.db_manager, os.getenv('OPENAI_API_KEY'))
         self.resume_path = os.getenv('BASE_RESUME_PATH')
-        self.stdscr = stdscr
         self.table_display = MatchingTableDisplay(self.stdscr, self.db_path)
         self.total_ai_job_recommendations = self.table_display.fetch_total_entries()
+        self.processed_listings_count = self.db_manager.fetch_processed_listings_count()
         self.total_listings = self.get_total_listings()
         env_limit = 0 if os.getenv('COMMANDJOBS_LISTINGS_PER_BATCH') is None else os.getenv('COMMANDJOBS_LISTINGS_PER_BATCH')
         self.listings_per_request = max(int(env_limit), 10)
@@ -66,16 +68,18 @@ def __init__(self, stdscr, logger):
             resume_menu = "📄 Edit resume"
             find_best_matches_menu = f"🧠 Find best matches for resume with AI (will check {self.listings_per_request} listings at a time)"
 
-        db_menu_item = f"💾 Navigate jobs in local db ({self.total_listings} listings)"
+        total_processed = f'{self.processed_listings_count} processed with AI so far'
+        db_menu_item = f"💾 Navigate jobs in local db ({self.total_listings} listings, {total_processed})"
         ai_recommendations_menu = "😅 No job matches for your resume yet"
         if self.total_ai_job_recommendations > 0:
-            ai_recommendations_menu = f"✅ AI found {self.total_ai_job_recommendations} listings match your resume and job preferences"
+            ai_recommendations_menu = f"✅ {self.total_ai_job_recommendations} recommended listings, out of {total_processed}"
 
         self.menu_items = [resume_menu, "🕸  Scrape \"Ask HN: Who's hiring?\"",
                            db_menu_item, find_best_matches_menu, 
                            ai_recommendations_menu]  # New menu option added
         self.current_row = 0
-        self.setup()
+        self.display_splash_screen()
+        self.run()
 
     async def process_with_gpt(self):
         try:
@@ -91,9 +95,9 @@ def update_ui(message):
             self.logger.exception("Failed to process listings with GPT: %s", str(e))
         finally:
             # Update menu items and redraw the menu after scraping is done
+            self.processed_listings_count = self.db_manager.fetch_processed_listings_count()
             self.update_menu_items()
             self.stdscr.refresh()
-            self.stdscr.getch()  # Wait for any key press after completion
 
     def read_resume_from_file(self):
         try:
@@ -102,14 +106,21 @@ def read_resume_from_file(self):
         except FileNotFoundError:
             return ""
 
-    def setup(self):
+    def setup_ncurses(self):
         curses.curs_set(0)  # Turn off cursor visibility
-        self.display_splash_screen()
         self.stdscr.keypad(True)  # Enable keypad mode
         curses.start_color()
         curses.init_pair(1, curses.COLOR_BLACK, curses.COLOR_CYAN)
         curses.init_pair(2, curses.COLOR_BLACK, curses.COLOR_WHITE)
-        self.run()
+        curses.init_pair(3, curses.COLOR_WHITE, curses.COLOR_BLUE)  # Highlight color
+        curses.init_pair(4, curses.COLOR_BLACK, curses.COLOR_WHITE)  # Highlight headers color
+        curses.init_pair(5, curses.COLOR_WHITE, curses.COLOR_MAGENTA)  # Highlight headers color
+        curses.init_pair(6, curses.COLOR_RED, curses.COLOR_BLACK)  # Highlight headers color
+        curses.init_pair(7, curses.COLOR_GREEN, curses.COLOR_BLACK)
+        curses.init_pair(8, curses.COLOR_YELLOW, curses.COLOR_BLACK)
+        curses.init_pair(9, curses.COLOR_BLUE, curses.COLOR_BLACK)
+        curses.init_pair(10, curses.COLOR_MAGENTA, curses.COLOR_BLACK)
+        curses.init_pair(11, curses.COLOR_RED, curses.COLOR_BLACK)
 
     def display_splash_screen(self):
         splash_text = [
@@ -130,15 +141,23 @@ def display_splash_screen(self):
         ]
         self.stdscr.clear()
         max_y, max_x = self.stdscr.getmaxyx()
-        self.stdscr.attron(curses.color_pair(6))
-        for i, line in enumerate(splash_text):
-            # Calculate the starting position for each line to be centered
-            start_x = max(0, (max_x - len(line)) // 2)
-            self.stdscr.addstr(i + (max_y - len(splash_text)) // 2, start_x, line)
+
+        # Repeat base animation 3 times
+        for i in range(0, 3):
+            # Loop through color pairs 7 to 11
+            # defined inside setup_ncurses()
+            for color in range(7, 12):
+                self.stdscr.attron(curses.color_pair(color))
+                for i, line in enumerate(splash_text):
+                    # Calculate the starting position for each line to be centered
+                    start_x = max(0, (max_x - len(line)) // 2)
+                    self.stdscr.addstr(i + (max_y - len(splash_text)) // 2, start_x, line)
+                self.stdscr.refresh()
+                # 100ms per color
+                curses.napms(100)
+                self.stdscr.attroff(curses.color_pair(color))
+        self.stdscr.clear()
         self.stdscr.refresh()
-        # Display the splash screen for 500 milliseconds
-        curses.napms(1000)
-        self.stdscr.attroff(curses.color_pair(6))
 
     def draw_title(self, title="Command Jobs"):
         max_y, max_x = self.stdscr.getmaxyx()
@@ -149,7 +168,6 @@ def draw_title(self, title="Command Jobs"):
         self.stdscr.addstr(1, 0, "-" * max_x)
 
     def draw_menu(self):
-        self.stdscr.clear()
         self.draw_title()
         h, w = self.stdscr.getmaxyx()
         for idx, item in enumerate(self.menu_items):
@@ -192,10 +210,11 @@ def update_menu_items(self):
             find_best_matches_menu = f"🧠 Find best matches for resume with AI (will check {self.listings_per_request} listings at a time)"
 
         # Update menu items with the new counts
-        db_menu_item = f"💾 Navigate jobs in local db ({self.total_listings} listings)"
+        total_processed = f'{self.processed_listings_count} processed with AI so far'
+        db_menu_item = f"💾 Navigate jobs in local db ({self.total_listings} listings, {total_processed})"
         ai_recommendations_menu = "😅 No job matches for your resume yet"
         if self.total_ai_job_recommendations > 0:
-            ai_recommendations_menu = f"✅ AI found {self.total_ai_job_recommendations} listings match your resume and job preferences"
+            ai_recommendations_menu = f"✅ {self.total_ai_job_recommendations} recommended listings, out of {total_processed}"
 
         # Update the relevant menu items
         self.menu_items[0] = resume_menu
@@ -211,8 +230,9 @@ def update_menu_items(self):
     # eg. first option (0): self.menu_items[0] = resume_menu 
     # = "Create or replace base resume"
     def execute_menu_action(self):
+        exit_message = ''
         if self.current_row == 0:  # Create or replace base resume
-            self.manage_resume(self.stdscr)
+            exit_message = self.manage_resume(self.stdscr)
         elif self.current_row == 1:  # Scrape "Ask HN: Who's hiring?"
             self.start_scraping_with_status_updates()
         elif self.current_row == 2:  # Navigate jobs in local db
@@ -222,11 +242,15 @@ def execute_menu_action(self):
         elif self.current_row == 4:  # Index of the new menu option
             self.table_display.draw_table()
         self.stdscr.clear()
+        if exit_message != '':
+            self.update_status_bar(exit_message)
 
-    def display_text_with_scrolling(self, header, lines, resume_path):
+    def display_text_with_scrolling(self, header, lines):
         curses.noecho()
         max_y, max_x = self.stdscr.getmaxyx()
         offset = 0  # How much we've scrolled
+        resume_updated = False
+        new_lines = ''
 
         while True:
             self.stdscr.clear()
@@ -249,10 +273,12 @@ def display_text_with_scrolling(self, header, lines, resume_path):
                 if offset > 0:
                     offset -= 1
             elif key in [ord('r'), ord('R')]:
-                lines = self.capture_text_with_scrolling()
-                with open(resume_path, 'w') as file:
-                    file.writelines(lines)
+                new_lines = self.capture_text_with_scrolling()
+                if len(new_lines) > 0:
+                    resume_updated = new_lines != lines
                 break
+
+        return resume_updated
 
     def get_total_listings(self):
         """Return the total number of job listings in the database."""
@@ -266,24 +292,25 @@ def get_total_listings(self):
     def manage_resume(self, stdscr):
         curses.echo()
         resume_path = os.getenv('BASE_RESUME_PATH')
+
+        resume_updated = False
+        exit_message = 'Resume not updated'
 
         if os.path.exists(resume_path):
             with open(resume_path, 'r') as file:
                 lines = file.readlines()
 
             header = "Base Resume (Press 'q' to go back, 'r' to replace):"  # Use a separator for clarity
-            self.display_text_with_scrolling(header, lines, resume_path)
+            resume_updated = self.display_text_with_scrolling(header, lines)
         else:
-            # Adjust the prompt position in capture_text_with_scrolling if needed
-            input_lines = self.capture_text_with_scrolling()
-            with open(resume_path, 'w') as file:
-                file.writelines(input_lines)
-            stdscr.clear()
-            self.draw_title("Resume saved. Press any key to continue...")  # Redraw title after clearing
-            stdscr.getch()
-            self.update_menu_items()  # Redraw the menu with updated items
+            resume_updated = self.capture_text_with_scrolling()
+
+        if resume_updated:
+            exit_message = f"Resume saved to {self.resume_path}"
+
+        return exit_message
 
-    def update_scraping_status(self, text):
+    def update_status_bar(self, text):
         max_y, max_x = self.stdscr.getmaxyx()
         # Ensure the status text will not overflow the screen width
         status_text = text[:max_x - 3]
@@ -300,19 +327,19 @@ def update_scraping_status(self, text):
     def start_scraping_with_status_updates(self):
         # Create a queue to receive the result from the scraping thread
         result_queue = Queue()
-        # Pass self.update_scraping_status as the update function to HNScraper
+        # Pass self.update_status_bar as the update function to HNScraper
         self.scraper = HNScraper(self.db_path)  # Initialize the scraper
         start_url = os.getenv('HN_START_URL')  # Starting URL
         scraping_thread = threading.Thread(target=self.scraper.scrape_hn_jobs, args=(
-            start_url, self.stdscr, self.update_scraping_status, self.scraping_done_event, result_queue))
+            start_url, self.stdscr, self.update_status_bar, self.scraping_done_event, result_queue))
         scraping_thread.start()
         # Call this method after the scraping is done
         self.scraping_done_event.wait()  # Wait for the event to be set by the scraping thread
         # Retrieve the result from the queue
         new_listings_count = result_queue.get()  # This will block until the result is available
         self.update_menu_items()  # Update the menu items after scraping
         self.draw_menu()  # Redraw the menu with updated items
-        self.update_scraping_status(f"Scraping completed {new_listings_count} new listings added")
+        self.update_status_bar(f"Scraping completed {new_listings_count} new listings added")
         self.stdscr.refresh()  # Refresh the screen to show the updated menu
         self.stdscr.getch()  # Wait for any key press after completion
         self.scraping_done_event.clear()  # Clear the event for the next scraping operation
@@ -380,7 +407,14 @@ def capture_text_with_scrolling(self):
                 x += 1
             self.stdscr.refresh()
 
-        return ''.join(text)
+        input_lines = ''.join(text)
+        if text != []:
+            with open(self.resume_path, 'w') as file:
+                file.writelines(input_lines)
+
+        curses.curs_set(0) # hide cursor again
+
+        return input_lines
 
 # Ensure logging is configured to write to a file or standard output
 logging.basicConfig(filename='application.log', level=logging.DEBUG, format='%(asctime)s %(levelname)s %(name)s %(message)s')