Cargo.toml

@@ -7,7 +7,7 @@ repository = "https://github.com/chotchki/feophant"
 keywords = ["sql", "database", "feophant"]
 categories = ["database-implementations"]
 license = "AGPL-3.0-or-later"
-version = "0.8.0"
+version = "0.9.0"
 edition = "2018"
 exclude = [
     "docs/*",
@@ -45,11 +45,13 @@ bitflags = "1.2.1"
 bytes = "1"
 futures = "0.3"
 hex-literal = "0.3.1"
-lru = "0.6.6"
-nom = "6"
+moka = { version = "0.6", features = ["future"] }
+#nom = "7"
+nom = { path = "/Users/chotchki/workspace/nom"}
+nom-supreme = "0.6.0"
 log = "0.4"
 simplelog = "^0.10.0"
-tokio = { version = "1", features = ["full"] }
+tokio = { version = "^1.12", features = ["full"] }
 tokio-stream = "0.1"
 tokio-util = { version = "0.6.6", features = ["codec"] }
 thiserror = "1.0"

README.md

@@ -2,15 +2,18 @@
 
 A SQL database server written in Rust and inspired by PostreSQL.
 
-Just a toy for the moment, but I'm actively working to fix that!
+We now have support for persistent storage! Not crash safe but I'm getting there!
 
 [![Latest Build][build-badge]][build-url]
 [![codecov][codecov-badge]][codecov-url]
+[![dependency status][dep-badge]][dep-url]
 
 [build-badge]: https://github.com/chotchki/feophant/actions/workflows/test_source_coverage.yaml/badge.svg
 [build-url]: https://github.com/chotchki/feophant/actions/workflows/test_source_coverage.yaml
 [codecov-badge]: https://codecov.io/gh/chotchki/feophant/branch/main/graph/badge.svg?token=6JV9391LY0
 [codecov-url]: https://codecov.io/gh/chotchki/feophant
+[dep-badge]: https://deps.rs/repo/github/chotchki/feophant/status.svg
+[dep-url]: https://deps.rs/repo/github/chotchki/feophant
 
 [Website](https://feophant.com)
 
@@ -20,137 +23,17 @@ Launch the server
 `./feophant`
 
 Lauch a postgres client application to test
+`./pgbench -i -h 127.0.0.1 -p 50000`
 `./pgbench -h 127.0.0.1 -p 50000`
 `./psql -h 127.0.0.1 -p 50000`
 
 Benchmark to aid in profiling
 `cargo instruments --bench feophant_benchmark -t time`
 
 ## What works user facing
-You can currently start the server, connect to it and have it throw tons of errors. To support more there is a ton of infrastructure required to wire up next steps.
-
-## Current TODO List - Subject to constant change!
-
-**TODO**
-
-Add support for defining a primary key on a table. This implies the following functionality:
-* Index support through the stack down to the page level.
-* The concept of unique indexes.
-* Transactional support for indexes.
-* Failure of a statement on constraint violation. Unsure if I'll end up with a general constraint system from this.
-
-Based on reading this really means implementing Btree indexes. They don't seem to be that bad to understand/implement.
-
-First and most important question, how should the index layers work?
-    Are they transactional? (I don't think so until I implement a visability map)
-    How should the low level layer function? 
-        Should I have an Index config struct I pass around or just a table + columns + unique or not + type
-        Index Config it is
-
-Index Manager -> for a given table
-IO Manager -> Handle Page Load / Store / Update
-
-Implemented the formats but I think I need to add locking to the I/O manager.
-    At a minimum I need to support a get for update, update and release lock.
-    I'm not sure I understand how this should work :(. I think need to commit to another layer.
-
-Implemented stronger page checking to make sure we don't store something that won't work on the file system.
-
-Back to indexes for now. I need to make a decision on how to handle them hitting the file system.
-    Postgres uses a series of OIDs to map onto disk.
-
-    I've been using uuids, I think I'm going to continue that. That would also solve the postgres fork approach.
-
-I'll have to switch IOManager to use uuid instead of Table as a key. Upside, I'm basically already doing that. (done)
-
-Next up implementing the index manager to add entries to the index.
-
-I'm having a hard time figuring this out, I might work to do the operations on the tree before I keep messing with the serialization protocols. I'm just worries they are directly linked.
-
-Got further into the index manager. Unfortunately I need a lock manager to let it even pass the smell test. Time to go on a wild goose chase again! (This project is great for someone with ADHD to have fun on!)
-
-The lock manager design/code is done but I'm not happy with using a rwlock to protect a tag. I really want to have the lock protect the content but that needs a way for me to support writeback. I think I need to build out two more things, a WAL mechanism and a buffer manager.
-
-I guess I need to commit to doing this for reals. However I am worried about reaching a point of partially working for a while like when I did the type fixing. We'll see how this goes.
-
-For now, the index implementation is now on hold until I get an integrated I/O subsystem and a stubbed out WAL.
-
-The real I/O subsystem has been written and tested. About to integrate it.
-Real integration tests using a non feophant postgres rust driver are also working.
-
-I/O subsystem integrated but I'm not happy with the performance. I'm debating a caching layer above the file_manager. I need to get some benchmarks together so I know if I'm actually helping.
-
-**TODO**
-
-
-Implement where clauses, will likely need to have to start tracing columns from analyizing through to later stages.
-
-**TODO**
-
-Implement support for running a fuzzer against the code base to ensure we are keeping the code at a high quality.
-
-
-
-**TODO**
-
-Implement delete for tuples
-
-**TODO**
-
-pgbench setup can run successfully, in memory
-
-**TODO**
-
-Ensure data about table structures is thread safe in the face of excessive Arc usage.
-
-See where I can pass read only data by reference instead of uisng Arc everywhere
-
-**TODO**
-
-Support a row with more than 4kb of text in it.
-
-**TODO**
-
-Implement sorting.
-
-**TODO**
-
-Implement column aliasing
-
-**TODO**
-
-Implement subselect.
-
-**TODO**
-
-Implement Updates.
-
-**TODO**
-
-Did some reading on how the buffer manager works and my implementation seems to be firmly in the right direction. Take that knowledge and implement persistence
-
-**1.0 Release Criteria**
-
-* pgbench can run successfully
-* ~~Pick a new distinct name, rename everything~~ Done
-* Pick a license
-* Setup fuzz testing
-* Persist to disk with moderate crash safety
-* Be prepared to actually use it
-
-
-### Longer Term TODO
-
-This is stuff that I should get to but aren't vital to getting to a minimal viable product.
-* Right now the main function runs the server from primitives. The Tokio Tower layer will probably do it better.
-* The codec that parses the network traffic is pretty naive. You could make the server allocate 2GB of data for a DDOS easily.
-* * We should either add state to the codec or change how it parses to produce chunked requests. That means that when the 2GB offer is reached the server can react and terminate before we accept too much data. Its a little more nuanced than that, 2GB input might be okay but we should make decisions based on users and roles.
-* There is an extension that removes the need to lock tables to repack / vaccum. Figure out how it works!
-* * https://github.com/reorg/pg_repack
-* Investigate if the zheap table format would be better to implement.
-** Until I get past a WAL implementation and planner costs I don't think its worth it.
-** Since I extended the size of transaction IDs, I probably have a larger issue on my hands than normal postgres.
-*** Reading into the zheap approach I'm thinking that I might have some space saving options availible for me. In particular if a tuple is frozen so its always availible I could remove the xmin/xmax and pack more into the page. Need more thinking however my approach of questioning the storage efficency of each part of data seems to be worth it.
+* Connecting unauthenticated using a postgres client/driver. 
+* You can create tables, insert data and query single tables.
+* Data is persisted to disk, not crash safe and the on disk format is NOT stable.
 
 ## Postgres Divergance
 
@@ -174,7 +57,8 @@ Reasonable application error type creation: https://github.com/dtolnay/anyhow
 
 Library Errors: https://github.com/dtolnay/thiserror
 
-Rust's inability to treat enum variants as a type is a HUGE pain. I cheated and separated serialization from deserialization.
+Rust's inability to treat enum variants as a type is a HUGE pain. I end up having an enum to hold data and another enum to validate and match the sub type. The [RFC](https://github.com/rust-lang/rfcs/pull/2593) to fix this was postponed indefinately.
+
 
 ## Legal Stuff (Note I'm not a lawyer!)
 

SECURITY.md

@@ -10,4 +10,4 @@ No support is provided at this time. I am working directly off of main until I a
 
 ## Reporting a Vulnerability
 
-Since this project is still in toy status please open an issue or pull request for any security fixes.
+Since this project is still pre-alpha please open an issue or pull request for any security fixes.

benches/feophant_benchmark.rs

@@ -1,96 +1,62 @@
-use std::sync::Arc;
-use std::time::Duration;
-
 use criterion::BenchmarkId;
 use criterion::Criterion;
 use criterion::{criterion_group, criterion_main};
-
-use feophantlib::constants::Nullable;
+use feophantlib::engine::get_row;
+use feophantlib::engine::get_table;
+use feophantlib::engine::io::block_layer::file_manager2::FileManager2;
+use feophantlib::engine::io::block_layer::free_space_manager::FreeSpaceManager;
 use feophantlib::engine::io::row_formats::RowData;
-use feophantlib::engine::io::FileManager;
 use feophantlib::engine::io::RowManager;
-use feophantlib::engine::objects::types::BaseSqlTypes;
-use feophantlib::engine::objects::types::BaseSqlTypesMapper;
-use feophantlib::engine::objects::Attribute;
-use feophantlib::engine::objects::SqlTuple;
-use feophantlib::engine::objects::Table;
 use feophantlib::engine::transactions::TransactionId;
 use futures::pin_mut;
+use std::sync::Arc;
 use tempfile::TempDir;
 use tokio::runtime::Builder;
 use tokio_stream::StreamExt;
 
-fn get_table() -> Arc<Table> {
-    Arc::new(Table::new(
-        uuid::Uuid::new_v4(),
-        "test_table".to_string(),
-        vec![
-            Attribute::new(
-                "header".to_string(),
-                BaseSqlTypesMapper::Text,
-                Nullable::NotNull,
-                None,
-            ),
-            Attribute::new(
-                "id".to_string(),
-                BaseSqlTypesMapper::Uuid,
-                Nullable::Null,
-                None,
-            ),
-            Attribute::new(
-                "header3".to_string(),
-                BaseSqlTypesMapper::Text,
-                Nullable::NotNull,
-                None,
-            ),
-        ],
-    ))
-}
-
-fn get_row(input: String) -> SqlTuple {
-    SqlTuple(vec![
-            Some(BaseSqlTypes::Text(input)),
-            None,
-            Some(BaseSqlTypes::Text("blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah".to_string())),
-        ])
-}
-
 // Here we have an async function to benchmark
 async fn row_manager_mass_insert(row_count: usize) -> Result<(), Box<dyn std::error::Error>> {
     let tmp = TempDir::new()?;
     let tmp_dir = tmp.path().as_os_str().to_os_string();
 
     let table = get_table();
-    let fm = Arc::new(FileManager::new(tmp_dir.clone())?);
-    let rm = RowManager::new(fm);
+    let fm = Arc::new(FileManager2::new(tmp_dir.clone())?);
+    let fsm = FreeSpaceManager::new(fm.clone());
+    let rm = RowManager::new(fm, fsm);
 
     let tran_id = TransactionId::new(1);
 
     for i in 0..row_count {
         rm.clone()
-            .insert_row(tran_id, table.clone(), get_row(i.to_string()))
+            .insert_row(tran_id, &table, get_row(i.to_string()))
             .await?;
     }
 
     drop(rm);
 
     //Now let's make sure they're really in the table, persisting across restarts
-    let fm = Arc::new(FileManager::new(tmp_dir)?);
-    let rm = RowManager::new(fm);
+    let fm = Arc::new(FileManager2::new(tmp_dir.clone())?);
+    let fsm = FreeSpaceManager::new(fm.clone());
+    let rm = RowManager::new(fm, fsm);
 
     pin_mut!(rm);
     let result_rows: Vec<RowData> = rm
         .clone()
-        .get_stream(table.clone())
+        .get_stream(&table)
         .map(Result::unwrap)
         .collect()
         .await;
 
     assert_eq!(result_rows.len(), row_count);
-    for i in 0..row_count {
-        let sample_row = get_row(i.to_string());
-        assert_eq!(result_rows[i].user_data, sample_row);
-    }
+    result_rows
+        .iter()
+        .enumerate()
+        .take(row_count)
+        .map(|(i, row)| {
+            let sample_row = get_row(i.to_string());
+            assert_eq!(row.user_data, sample_row);
+        })
+        .for_each(drop);
 
     Ok(())
 }

docs/_config.yml

@@ -1,6 +1,6 @@
 theme: jekyll-theme-minimal
 title: FeOphant
 description: A database server written in Rust and inspired by PostreSQL
-logo: /feophant_logo.svg
+logo: /feophant_logo_v2.svg
 show_downloads: false
 google_analytics: G-GBPHN9NF6R