Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PySpark lance package #1466

Open
wjones127 opened this issue Oct 25, 2023 · 2 comments
Open

PySpark lance package #1466

wjones127 opened this issue Oct 25, 2023 · 2 comments
Labels
enhancement New feature or request python

Comments

@wjones127
Copy link
Contributor

While a proper implementation integrating at the JVM level (#541) will take time, we can create a quick prototype by wrapping pylance into PySpark functions.

Minimal scope:

  • Read (no pushdown)
  • Append, Overwrite, Create

If there is an easy way to enable read pushdown, we can add that to scope. But I don't see any extension API from Spark that allows that.
@wjones127 wjones127 added enhancement New feature or request python labels Oct 25, 2023
@jiaoew1991
Copy link

Is there any progress on this issue?

@wjones127
Copy link
Contributor Author

I haven't yet work on this. So no progress yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request python
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jiaoew1991 @wjones127