This repo is an experiment in trying to connect the spark-connect-rs client to the spark-connect server created by sail from lakehq.
sail is using datafusion behind the spark-connect server to translate a spark plan into a datafusion plan. The spark client for spark-connect-rs is written in rust, so this is a rust client sending a spark plan to a spark connect server also written in rust.
Start the python venv with uv and install the dependencies
uv venv && source .venv/bin/activate
Start the sail server with
sail spark server --port 50051
Run the SQL spark program
cargo run --bin sql