Arrow2 merge avro

[Igosuki]

Rationale for this change

Merge latest of arrow-datafusion, with latest arrow2 that still has RecordBatch

What changes are included in this PR?

Support for latest datafusion changes : timezone, decimal in more places, stddev, variance
Dictionary has a third value which is whether the dict is sorted
Pyo3 integration changed because arrow2 uses ffi directly
Suddenly serializing ipc requires ipc fields, I put empty ipc fields where I couldn't get the schema so this may require a fix
Avro support fully migrated to arrow
simd integration is not available on stable, this requires a PR fixed on arrow2
arrow2 DataType and ScalarValue don't implement display

Some code, notably in tests could be further refactored due to better ergonomics in arrow2.

Are there any user-facing changes?

As required, no API changes externally for datafusion, however

[Igosuki]

I haven't made all tests pass so this is a draft

[Igosuki]

Got this nicety when running miri on nightly :

test tests::ballista_round_trip::q1 ... error: unsupported operation: can't call foreign function: epoll_create1
    --> /home/geps/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.7.14/src/sys/unix/selector/epoll.rs:34:9
     |
34   |         syscall!(epoll_create1(flag)).map(|ep| Selector {
     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't call foreign function: epoll_create1

cf rust-lang/miri#602

[houqp]

Thank you @Igosuki ! I will take a close look at it tonight and merge it. Then 3 of us can collaborate on fixing the failing tests through follow up PRs.

simd integration is not available on stable, this requires a PR fixed on arrow2

I believe SIMD requiring nightly is expected?

[Igosuki]

@houqp Yeah I mentioned it on another PR and got the answer, I didn't think it was required

[houqp]

@Igosuki I have pushed your changes into my arrow2_merge branch as is. Let's collaborate on this branch to fix the remaining tests. cc @yjshen

[jorgecarleitao]

Thanks a lot for this!

pass None to the IPC fields to cause arrow2 to derive defaults for them. They are only relevant for dictionary arrays. This is in preparation to automatically re-use dictionary arrays.

The pyo3 was not added because pyo3 on arrow2 causes all dependencies to require a specific version of it (polars, datafusion-python)

[Igosuki]

@jorgecarleitao thanks for clarifying, I hope https://github.com/houqp/arrow-datafusion/blob/arrow2_merge/datafusion/src/pyarrow.rs is a good enough replacement, I used the code in your integration tests.

[houqp]

@Igosuki @yjshen I fixed all the decimal tests, so we are now down to 8 test failures, mostly file format related.

[houqp]

@jorgecarleitao i noticed the IPC stream writer's write API is not consistent with what we have for the file writer: https://github.com/jorgecarleitao/arrow2/blob/a886e35748b2813e6c5a122cf9603a3e59f9f80e/src/io/ipc/write/stream.rs#L58. Is this expected? It takes a slice instead of option for the ipc field argument.

[jorgecarleitao]

@jorgecarleitao i noticed the IPC stream writer's write API is not consistent with what we have for the file writer: https://github.com/jorgecarleitao/arrow2/blob/a886e35748b2813e6c5a122cf9603a3e59f9f80e/src/io/ipc/write/stream.rs#L58. Is this expected? It takes a slice instead of option for the ipc field argument.

It is an API bug. It should be Option<&[]> like the other. Sorry about that.