[Website] Add "Data wants to be free" #586
Conversation
|
I did not finish annotating the binary blobs, I'll finish that if we're OK with the scheme I've chosen so far |
| Instead of putting lengths of values everywhere, Arrow groups values of the same column (and hence same type) together, so it just needs the length of the buffer. Strings do still require a length per value, but the overhead isn’t added where it isn’t otherwise needed. And nullability is instead stored in a bitmap, which is omitted if there aren’t any NULL values in the first place, saving space. Because of that, more rows of data doesn’t increase the overhead; instead, the more data you have, the less you pay\! | ||
|
|
||
| Even the header isn’t actually the disadvantage it looks like. The header contains the schema, which makes the data stream self-describing. With PostgreSQL, you need to get the schema from somewhere else. So we aren’t making an apples-to-apples comparison in the first place: PostgreSQL still has to transfer the schema, it’s just not part of the “binary format” that we’re looking at here. | ||
| Meanwhile, there’s actually a more insidious problem with PostgreSQL we’ve overlooked so far: alignment. Remember that 2 byte field count at the start of every row? Well, that means all the 4 byte integers after it are now unaligned…so you can’t use them without copying them (or doing a very slow unaligned load). Arrow, on the other hand, strategically adds some padding (overhead) to align the data, and lets you use little-endian or big-endian byte order depending on your data. And Arrow doesn’t apply expensive encodings to the data that require further parsing; there’s just optional compression that can be enabled if it suits your data[^2]. So **you can use Arrow data as-is without having to parse every value**. |
There was a problem hiding this comment.
Hmm, it's a been a long time since unaligned loads were "very slow". You might get some overhead from time to time, especially if a cache line or a page is straddled, but I'm not sure the difference is really important.
Also, the alignment really only comes into play if the file is being memory-mapped.
There was a problem hiding this comment.
Though we have found issues in the past with Rust since Rust gets very upset with unaligned data buffers.... maybe mention that?
There was a problem hiding this comment.
Rust vs unaligned data is more of a "source-level" issue AFAIU. I suppose you just have to be careful and use something like https://doc.rust-lang.org/std/ptr/fn.read_unaligned.html (as we also do in C++, actually).
There was a problem hiding this comment.
Is that true for all architectures, and isn't it still undefined behavior to have a pointer to an unaligned value? (In which case either you have to rewrite your code to memcpy explicitly, or parse the values...)
There was a problem hiding this comment.
Hmm, that's a good question. From what I understand:
- At least x86 and ARM allow unaligned accesses; SPARC doesn't (but who cares nowadays? :-)); not sure about RISC-V and POWER
- on the source language side, it's true that dedicated constructs may be required (such as the memcpy trick, which is typically optimized by the compiler into a regular or unaligned load instruction)
In any case, I think we can relax the wording in that sentence a bit to explain the inconvenience more clearly and without overblowing the issue:
Well, that means all the 4 byte integers after it are now unaligned, which then require care to handle properly (dedicated constructs can be required at the source language level - such as C++ or Rust - and performance may suffer depending on the CPU)
There was a problem hiding this comment.
I adjusted the wording here.
|
|
||
| # PostgreSQL vs Arrow: Data Serialization | ||
|
|
||
| Let’s compare the [PostgreSQL binary |
There was a problem hiding this comment.
Is the PostgreSQL binary format different from the data serialization format used in the PostgreSQL wire protocol? If it is, I'm not sure the comparison between it and Arrow is fair. If it is, I think a couple sentences to draw the user in would be good.
When you connect to a PostgreSQL database and execute a query, PostgreSQL sends the result back to you using the PostgreSQL wire protocol. The data in your query result is encoded within this protocol using the PostgreSQL binary format.
(Note: The above could be false but it's the basic thing I was after)
There was a problem hiding this comment.
Also, for where we mention "PostgreSQL wire protocol" lower down, I think we should add a link to each instance, such as to https://www.postgresql.org/docs/current/protocol.html.
There was a problem hiding this comment.
In this context, I'm pretty sure the PostgreSQL binary format is the PostgreSQL wire protocol
| 000003c8: 00 01 00 00 41 52 52 4f ....ARRO | ||
| 000003d0: 57 31 W1</code></pre></div></div> | ||
|
|
||
| Arrow looks quite…intimidating…at first glance. There’s a giant header, and lots of things that don’t seem related to our dataset at all, plus mysterious padding that seems to exist solely to take up space. But the important thing is that **the overhead is fixed**. Whether you’re transferring one row or a billion, the overhead doesn’t change. And unlike PostgreSQL, **no per-value parsing is required**. |
There was a problem hiding this comment.
I wonder if we couldn't come up with a graphical demonstration of the overhead. As I'm reading this, I find I want a chart or table or something to compare statements like the one you've made here about overhead.
There was a problem hiding this comment.
A summary chart makes sense.
There was a problem hiding this comment.
I started to tinker around with a graphical comparison of overhead by comparing PostgreSQL binary files to Arrow IPC files for various rows counts and schemas and the result was not very interesting. For the schema in this post (bigint, text, bigint), plotting file size against number of rows/records on a log10 scale,
By 100 records, Arrow and Postgres are close and by 1000 records the overhead increases at a constant factor. Feel free to resolve this conversation unless you think of anything.
|
I've been distracted with other work but I'll try to integrate the feedback and replace the missing diagrams in the next week or so, thanks everyone! |
Co-authored-by: Bryce Mecum <[email protected]>
|
I've mostly addressed things here; I do plan to take another pass and shorten the prose (and probably de-emphasize size further vs other features) |
| layout: post | ||
| title: "Data wants to be free: fast data exchange with Apache Arrow" | ||
| description: "" | ||
| date: "2025-02-04 00:00:00" |
There was a problem hiding this comment.
We should change the date here (and in the filename) to whatever day this actually posts.
There was a problem hiding this comment.
I set the date to the 28th
Co-authored-by: Ian Cook <[email protected]>
|
Should we make the title be in title case? Data Wants to Be Free: Fast Data Exchange with Apache Arrow |
|
Fixed |
No description provided.