Add Spark data source for lance

[LiWeiJie]

No description provided.

[LiWeiJie]

Hope to support lance in pyspark's spark data source

[andrei-ionescu]

Is there a Scala implementation for Lance format?

[changhiskhan]

Is there a Scala implementation for Lance format?

Not yet. We're still debating how to do it long term. The data will come out as Arrow which has a JVM impl already.

If you're interested in hacking on one let us know! [email protected] would love to collaborate

[zhenyu]

+1 for the feature!

[wjones127]

TBD: DataSourceV2 or TableProvider?

[rjurney]

This would be really cool... want to use Lance with PySpark.

[SaintBacchus]

Lance spark connector can already be used in our case. But now it misses some docs about how to use that. I will write it soon.

[rjurney]

I couldn't find a lance spark connector. Can you post a URL and then I guess this ticket can be closed?

[SaintBacchus]

Now there are no pre-build jars for lance spark connectors. If you want to use it now, you should build it yourself.

I submit a new PR for how to build the java package and use the spark connector #3433.
But I think it is still hard to understand. Maybe you can wait a few days until the new version of Lance is released. We will publish the pre-build jars into the Maven repo and write some docs about how to use the spark connectors.