LLM Data Ingestion into Editable Table

[MelvinSninkle]

Title: Ingest LLM Data into Editable Table
Description: Develop a data ingestion pipeline to store new headlines and associated metadata in a structured, easily editable table.
Dependencies: #4
Tasks:
1. Create a database table with fields: URL, old headline, new headline, model used, prompt identifier, date, and time of action.
2. Build an ingestion pipeline to store LLM-generated headlines and metadata.
3. Implement edit functionality for easy updates to any field.
4. Include logging for all data ingestion activities.
Acceptance Criteria:
• Database table is correctly set up with all specified fields.
• Data is ingested accurately and is easily editable.
• Successful logging of all ingestion activities and changes.
Priority: High
Labels: Database, Backend, Data Ingestion, MVP

[Melvillian]

LGTM 👍

Which Database should we use? I think Postgres is the simplest choice because it's so widely used, and has generic-enough text querying capabilities to do what we want.

To make it simpler we can use a hosted solution for Postgres. I've heard good things about Neon, but we could also go with Supabase which I think is more popular.

My suggestion is to go with Supabase using Postgres

[MelvinSninkle]

I am good with your suggestion on the tech side. Do you know what we need to get us off the ground monetarily?

[Melvillian]

Looks like $25/month should get us to a point where if we need to pay more than $25, that is a good problem to have!

https://supabase.com/pricing

[MelvinSninkle]

I can get that set up

…

On Thu, Oct 24, 2024 at 8:30 AM Alex Melville ***@***.***> wrote: Looks like $25/month should get us to a point where if we need to pay more than $25, that is a good problem to have! https://supabase.com/pricing image.png (view on web) <https://github.com/user-attachments/assets/e629deb9-2331-47f4-abea-62f596ed9d3a> — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BL5APYXS5LOZEDVUWOWTF43Z5EHAPAVCNFSM6AAAAABQI6QTXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZVGYYTGNBYGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>