[Website]: DataFusion 26-34 blog post

[alamb]

Closes apache/datafusion#6780

This blog post describes DataFusion over the last 6 months, DataFusion 26 to 34.

If anyone has time to pitch in and look up links or help with the language that would be most apprecaited

[alamb]

I hope to publish this post at the end of the week -- around Jan 19

[andygrove]

This looks excellent. Thank you @alamb

[alamb]

@jayzhan211 is there any other improvements you think we should call out about struct/array support over the last 6 months?

[jayzhan211]

nope. I think all of them are new features compare to 6 months before.

[alamb]

@ozankabak are there any plans you and your team may have that you want to share publically?

[ozankabak]

We don't have anything to share just yet on features we will contribute in 2024 (but there will be many!). We will probably have something to publish in a month or two.

[crepererum]

I think this plot is hard to read, mostly due to the wildly different numbers and because "execution time" often is an aggregate. For these kinds of plots, I see two possible ways to improve them:

• log axis: use a log axis for time, because improvements are often not linear deltas but factors and a log-space would account for that nicely. That would also make the wildly different numbers easier to read. Drawback: people don't read log space very well.
• relative factor: Only draw bars for DF v34 as a factor relative to v25 (which would be <1.0x in most cases) and on top of the bar (or at the base) print the seconds it took for v25. This tells the story of the change but also gives readers a baseline.

[alamb]

This is a good call --

I took a call creating this version. What do you think?

BTW source data: https://github.com/alamb/datafusion-duckdb-benchmark/tree/datafusion-25-34

Sheet https://docs.google.com/spreadsheets/d/1FtI3652WIJMC5LmJbLfT3G06w0JQIxEPG4yfMafexh8/edit#gid=530035076

[alamb]

I used this new chart. Here is what the page looks like rendered now:

[crepererum]

That's better. I think that the execution time (blue) shouldn't be a line plot because that type implies a connection between the neighboring points (like a time series where subsequent entries are indeed related). The linear interpolation between the measurements makes this even more misleading / "weird".

[alamb]

I couldn't figure out how to make Google sheets do what I wanted, so I eventually just made two charts so that one shows the overall magnitude and one shoes the relative improvement

I am sure we can do better if we spent more time on this, but I think it is good enough for now

[ozankabak]

We plan to write show-and-tell blog posts and videos that explain how one can use Datafusion in real-world use cases. We will try to partner with members of the community to create toy examples relating to their use cases and try to come up with demo scripts that offer guidance to others on how they can use Datafusion in similar contexts.

Maybe it could be a good idea to mention these upcoming show-and-tells as a near-future community growth effort.

[alamb]

Yes, this is great. I will include it

[alamb]

in 95158bb

[alamb]

BTW does any one have an example of writing to remote object storage (e.g. s3) handy that they could share so I can include it here?

@devinjdangelo do you have this setup ?

[devinjdangelo]

@alamb We have an example of writing to S3 here: https://github.com/apache/arrow-datafusion/blob/main/datafusion-examples/examples/external_dependency/dataframe-to-s3.rs

[devinjdangelo]

COPY table to 's3://my_bucket/my_prefix' should work in datafusion-cli so long as the credentials are set up. I'll verify real quick...

[devinjdangelo]

Ok confirmed with caveat. Using COPY directly to object store from datafusion cli does not work, but insert to external table does. We probably need to add special logic to datafusion-cli to make copy to object store to work directly. That would be a neat feature to add.

For now this works:

export AWS_SECRET_ACCESS_KEY=...
export AWS_ACCESS_KEY_ID=...
export AWS_DEFAULT_REGION=...
datafusion-cli
❯ create external table remote_table2(a int, b int) stored as parquet location 's3://dfcli-test-bucket2/';
0 rows in set. Query took 0.001 seconds.

❯ insert into remote_table2 values (1,2);
+-------+
| count |
+-------+
| 1     |
+-------+
1 row in set. Query took 0.272 seconds.
❯ select * from remote_table2;
+---+---+
| a | b |
+---+---+
| 1 | 2 |
+---+---+
1 row in set. Query took 0.131 seconds.

I see the parquet file in my s3 bucket as well.

[alamb]

Thanks @devinjdangelo -- I'll file a ticket about this in DataFusion later today

[devinjdangelo]

#8907 filed to make the COPY example work...

[alamb]

Ok confirmed with caveat. Using COPY directly to object store from datafusion cli does not work, but insert to external table does. We probably need to add special logic to datafusion-cli to make copy to object store to work directly. That would be a neat feature to add.

Thank you 🙏

[alamb]

Thank you everyone who helped with this project

[alamb]

The blog post is now published! https://arrow.apache.org/blog/2024/01/19/datafusion-34.0.0/