Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve SQLPage Notebook Storage Structure #147

Open
7 tasks
shah opened this issue Nov 16, 2024 · 0 comments
Open
7 tasks

Improve SQLPage Notebook Storage Structure #147

shah opened this issue Nov 16, 2024 · 0 comments
Assignees

Comments

@shah
Copy link
Contributor

shah commented Nov 16, 2024

Currently, all SQLPage TypeScript Notebooks results are stored in the sqlpage_files physical table. While functional, this approach limits flexibility in extending or dynamically generating SQLPage content.

Tasks

  1. Create the sqlpage_notebook_cell table with fields similar to code_notebook_cell.
  2. Update code in lib/std/notebook/sqlpage.ts to generate code in sqlpage_notebook_cell instead of sqlpage_files.
  3. Implement the sqlpage_files SQL view to select from sqlpage_notebook_cell and support dynamic content integration.
  4. Add Cookbook recipes which show how to ingest *.sqlpage.md into sqlpage_notebook_cell entries.
  5. Update documentation to reflect these changes.

Benefits

  • Uniformity: Aligns SQLPage notebooks with the flexible, state-based structure of code_notebook_cell.
  • Flexibility: Supports dynamic and external content integration through SQL views.
  • Interoperability: Leverages SQL for combining diverse data sources, making SQLPage content extensible.
  • Markdown Support: Streamlines the inclusion of Markdown-based content with frontmatter metadata.

Storage Shift:

  • Migrate the storage of SQLPage Notebooks to a new table, sqlpage_notebook_cell, similar to the code_notebook_cell structure. This will align SQLPage with existing notebook paradigms.

View Creation:

  • Replace the sqlpage_files table with a SQL view named sqlpage_files.
  • By default, this view will select rows from sqlpage_notebook_cell but can be modified by devs for their own purposes.

Dynamic Content Integration:

  • Enhance the sqlpage_files view to allow combining data from other sources (e.g., uniform_resource ingestion data).
  • Use SQL techniques like Common Table Expressions (CTEs), unions, and joins to include dynamically generated or additional content.

Markdown Frontmatter Support:

  • Enable the ingestion and conversion of SQLPage content from Markdown files (*.sqlpage.md).
  • Attributes like path, shell, and navigation can be stored as Markdown frontmatter and converted into rows within the sqlpage_files view.

Example Use Case: Markdown to SQLPage Integration
SQLPages stored in Markdown files with frontmatter metadata can be processed and included dynamically in the sqlpage_files view.
This type of file would be ingested as usual using surveilr ingest files or make its way to uniform_resource in any other way necessary. Then, the sqlpage_files view could contribute the content by joining from uniform_resource.

Example: A file example.sqlpage.md containing:

---
path: /example
shell: ignore
navigation: 
  menu: X
---
-- SQLPage code goes here

This would result in rows in sqlpage_notebook_cell, making it accessible via the sqlpage_files view.

@shah shah self-assigned this Nov 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant