#269 - charybdis database uploading

[bracyw]

Changes

STILL NEED TO ADD LIKE EVERYTHING. Just an mvp of the basic data flow.

Currently we have two gpt'ed service functions that dump local to csv and one that uploads it to the cloud database using prisma inserts both use batch querying with prisma to avoid to much memory consumption.

One function is also available to compare the local and cloud database (mostly for testing confirmation).

Notes

Any other notes go here

Test Cases

same amount of rows / tables in local as cloud

To Do

Any remaining things that need to get done

Server

• Allow for dev (with test db, probably just scylla idk) and production environment
• add tests or for now just add large seed in test db
• clean up service functions (get rid of any stuff, make more explicit, return good errors)
• controllers return helpful errors(allow for returning of good errors)
• add ability to check what step were on... make more resiliant against bs
• get into docker
• clean up dependencies
• make README

CLI

• add better feedback based on new api
• get into docker
• clean up dependencies
• make README

General

• update charybdis readme
• [ ]

Checklist

It can be helpful to check the Checks and Files changed tabs.
Please review the contributor guide and reach out to your Tech Lead if anything is unclear.
Please request reviewers and ping on slack only after you've gone through this whole checklist.

• All commits are tagged with the ticket number
• No linting errors / newline at end of file warnings
• All code follows repository-configured prettier formatting
• No merge conflicts
• All checks passing
• Screenshots of UI changes (see Screenshots section)
• Remove any non-applicable sections of this template
• Assign the PR to yourself
• No package-lock.json changes (unless dependencies have changed)
• Request reviewers & ping on Slack
• PR is linked to the ticket (fill in the closes line below)

Closes #269