Upgrade DataFusion to arrow/parquet 57.0.0 #17888

alamb · 2025-10-02T14:40:31Z

Which issue does this PR close?

Related to Release arrow-rs / parquet Major version 57.0.0 (October 2025) arrow-rs#7835
Closes Incorrect error message for decimal with scale while input value is out of bound #3666

Note while this PR looks massive, a large portion is display updates due to better display of Fields and DataTypes

Rationale for this change

Upgrade to the latest arrow

Also, there are several new features in arrow-57 that I want to be able to test including Variant, arrow-avro, and a new parquet metadata reader.

What changes are included in this PR?

Update arrow/parquet
Update prost
Update substrait
Update pbjson
Make API changes to avoid deprecated APIs

Are these changes tested?

By CI

Are there any user-facing changes?

New arrow

alamb · 2025-10-02T15:41:51Z

Many of the current failures are due because this used to work:

select arrow_cast('2021-01-01T00:00:00', 'Timestamp(Nanosecond, Some("-05:00"))'

or

SELECT arrow_cast(secs, 'Timestamp(Millisecond, None)') FROM t

After the arrow 57 upgrade it fails with errors like

statement error DataFusion error: Execution error: Unsupported type 'Timestamp\(Nanosecond, None\)'\. Must be a supported arrow type name such as 'Int32' or 'Timestamp\(ns\)'\. Error expected double quoted string for Timezone, got 'None'

# arrow_typeof_timestamp
query T
SELECT arrow_typeof(now()::timestamp)
----
Timestamp(ns)

I believe the problem is that the format of the timezone has changed into Timestamp(ns) and then the FromStr method doesn't handle that. I will work on filing an update

I think what we need to do is support both formats for backwards compatibility. I will work on an upstream issue

alamb · 2025-10-02T17:18:34Z

datafusion-examples/examples/flight/flight_client.rs


    // Create Flight client
-    let mut client = FlightServiceClient::connect("http://localhost:50051").await?;
+    let endpoint = Endpoint::new("http://localhost:50051")?;


This is due to new version of tonic

alamb · 2025-10-02T17:18:57Z

datafusion-examples/examples/flight/flight_server.rs


                // add an initial FlightData message that sends schema
                let options = arrow::ipc::writer::IpcWriteOptions::default();
+                let mut compression_context = CompressionContext::default();


Due to

Reuse zstd compression context when writing IPC arrow-rs#8405

alamb · 2025-10-02T17:19:23Z

datafusion/functions-aggregate-common/src/utils.rs


-            let validate =
-                T::validate_decimal_precision(new_value, self.target_precision);
+            let validate = T::validate_decimal_precision(


Due to this (get better messages)

[Decimal] Add scale argument to validation functions to ensure accurate error logging arrow-rs#8396

I'll add "Closes #3666" to the PR body 👍

alamb · 2025-10-02T17:22:13Z

datafusion/sqllogictest/test_files/array.slt

-List(Field { name: "item", data_type: List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) List(Field { name: "item", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })
-List(Field { name: "item", data_type: List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) List(Field { name: "item", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })
-List(Field { name: "item", data_type: List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) List(Field { name: "item", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })
+List(nullable List(nullable Int64)) List(nullable Float64) List(nullable Utf8)


Many of the diffs in this file are related to improvements in DataType display, tracked in this ticket

Improve Display for DataType arrow-rs#8351

I will try and call out individual changes when I see them. Lists are way nicer now:

Improve Display for DataType and Field arrow-rs#8290

alamb · 2025-10-02T17:22:38Z

datafusion/sqllogictest/test_files/array.slt

 05)--------ProjectionExec: expr=[]
 06)----------CoalesceBatchesExec: target_batch_size=8192
-07)------------FilterExec: substr(md5(CAST(value@0 AS Utf8View)), 1, 32) IN ([Literal { value: Utf8View("7f4b18de3cfeb9b4ac78c381ee2ad278"), field: Field { name: "lit", data_type: Utf8View, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }, Literal { value: Utf8View("a"), field: Field { name: "lit", data_type: Utf8View, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }, Literal { value: Utf8View("b"), field: Field { name: "lit", data_type: Utf8View, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }, Literal { value: Utf8View("c"), field: Field { name: "lit", data_type: Utf8View, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} } }])
+07)------------FilterExec: substr(md5(CAST(value@0 AS Utf8View)), 1, 32) IN ([Literal { value: Utf8View("7f4b18de3cfeb9b4ac78c381ee2ad278"), field: Field { name: "lit", data_type: Utf8View } }, Literal { value: Utf8View("a"), field: Field { name: "lit", data_type: Utf8View } }, Literal { value: Utf8View("b"), field: Field { name: "lit", data_type: Utf8View } }, Literal { value: Utf8View("c"), field: Field { name: "lit", data_type: Utf8View } }])


Due to

Improve Display for DataType and Field arrow-rs#8290

alamb · 2025-10-02T17:23:21Z

datafusion/sqllogictest/test_files/arrow_typeof.slt

 SELECT arrow_typeof(now()::timestamp)
 ----
-Timestamp(Nanosecond, None)
+Timestamp(ns)


Due to

Improve Display formatting of DataType::Timestamp arrow-rs#8425

I believe we'll need to update https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/data_types.md and https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/scalar_functions.md (the latter via updating the source docs in code) too, but can do in a follow up

Great call. Done in 1e8ddfa and f9606d8

alamb · 2025-10-02T17:23:41Z

datafusion/sqllogictest/test_files/arrow_typeof.slt


 ## Timestamps: Create a table

+statement ok


The timestamp format has changed (improved!) so let's also add tests for the new format

alamb · 2025-10-02T17:25:17Z

datafusion/substrait/Cargo.toml

 pbjson-types = { workspace = true }
 prost = { workspace = true }
-substrait = { version = "0.58", features = ["serde"] }
+substrait = { version = "0.59", features = ["serde"] }


since prost is updated, we also must update substrait

datafusion-cli/src/main.rs

alamb · 2025-10-03T20:24:04Z

datafusion/common/src/dfschema.rs

-        let expected = "Field { name: \"c0\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, \
-        Field { name: \"c1\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }";
-        assert_eq!(expected, arrow_schema.to_string());
+        insta::assert_snapshot!(arrow_schema.to_string(), @r#"Field { "c0": nullable Boolean }, Field { "c1": nullable Boolean }"#);


many many diffs are due to the changes in formatting of Fields and DataTypes (see below)

alamb · 2025-10-03T20:24:26Z

datafusion/core/tests/dataframe/dataframe_functions.rs

+    +----------------------+
+    | arrow_typeof(test.l) |
+    +----------------------+
+    | List(nullable Int32) |


the new display is much easier to read in my opinion

alamb · 2025-10-03T21:02:38Z

Ok, the tests are now looking good enough to test with the new thrift decoder

alamb · 2025-10-04T10:03:17Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1016-gcp #17~24.04.1-Ubuntu SMP Wed Sep 3 01:55:36 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/upgrade_arrow_57 (0cfb693) to 0f3cf27 diff using: tpch_mem
Results will be posted here when complete

alamb · 2025-10-23T16:39:12Z

This PR is now ready for review

Jefffrey

LGTM 👍

Jefffrey · 2025-10-24T03:58:33Z

datafusion/core/tests/parquet/filter_pushdown.rs

    // Can disable the cache even with filter pushdown by setting the size to 0. In this case we
-    // expect the inner records are reported but no records are read from the cache


nit: wording a bit off here, since it reads as

Can disable the cache even with filter pushdown by setting the size to 0. In this case we no records are read from the cache and no metrics are reported

Should be this maybe?

Can disable the cache even with filter pushdown by setting the size to 0. This results in no records being read from the cache and no metrics being reported

Great call -- done in 3a90c6e

Jefffrey · 2025-10-24T04:01:46Z

datafusion/functions-aggregate-common/src/utils.rs


-            let validate =
-                T::validate_decimal_precision(new_value, self.target_precision);
+            let validate = T::validate_decimal_precision(


I'll add "Closes #3666" to the PR body 👍

Jefffrey · 2025-10-24T04:03:27Z

datafusion/sqllogictest/test_files/arrow_typeof.slt

 SELECT arrow_typeof(now()::timestamp)
 ----
-Timestamp(Nanosecond, None)
+Timestamp(ns)


I believe we'll need to update https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/data_types.md and https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/scalar_functions.md (the latter via updating the source docs in code) too, but can do in a follow up

Jefffrey · 2025-10-24T04:04:39Z

datafusion/sqllogictest/test_files/dates.slt

-statement error DataFusion error: type_coercion\ncaused by\nError during planning: Cannot coerce arithmetic expression Timestamp\(Nanosecond, Some\("\+00:00"\)\) \+ Utf8 to valid types
+statement error
 select i_item_desc from test
 where d3_date > now() + '5 days';
+----
+DataFusion error: type_coercion
+caused by
+Error during planning: Cannot coerce arithmetic expression Timestamp(ns, "+00:00") + Utf8 to valid types


I thought the expected error comes before the query not after, for SLTs 🤔

I think it can be either (this change was created by running with --complete)

I believe when the message comes after the query it can contain multiple lines (aka if the error itself actually contains `\n')

alamb · 2025-10-26T11:34:02Z

cargo audit failure seems unrelated

cargo audit is failing with tokio-tar parses PAX extended headers incorrectly, allows file smuggling #18288

alamb · 2025-10-27T12:19:09Z

Merging up to get the fix for

cargo audit is failing with tokio-tar parses PAX extended headers incorrectly, allows file smuggling #18288

alamb · 2025-10-27T14:16:53Z

Alright, here we go!

Followup some doc fixes missed in #17888

….0 (#17866) ## Which issue does this PR close? - Closes #17865. ## What changes are included in this PR? Bump the `substrait` version to `v0.75.0` by bumping `substrait-rs` to `v0.60.0`. This PR was originally dependent on [this PR](#17888) to update the versions of some common dependencies, but that PR is now merged in. ## Are these changes tested? There are no tests here, but there is no change to any logic within datafusion. It is simply a bump in a dependency. Technically the public API does change, but as noted in the issue description, there is no change to internal logic because uri / urn from substrait plans are not used. ## Are there any user-facing changes? Yes. Previously substrait plans of spec version `v0.74.0` were accepted, and now `v0.75.0` is accepted. However, this is a backwards compatible change. The only difference is the inclusion of additional urn-based fields in substrait plans. In a later PR, the old uri-based fields will be dropped, which *will* be a breaking change. --------- Co-authored-by: Andrew Lamb <[email protected]>

## Which issue does this PR close? - Related to apache/arrow-rs#7835 - Closes apache#3666 Note while this PR looks massive, a large portion is display updates due to better display of Fields and DataTypes ## Rationale for this change Upgrade to the latest arrow Also, there are several new features in arrow-57 that I want to be able to test including Variant, arrow-avro, and a new parquet metadata reader. ## What changes are included in this PR? 1. Update arrow/parquet 2. Update prost 3. Update substrait 4. Update pbjson 5. Make API changes to avoid deprecated APIs ## Are these changes tested? By CI ## Are there any user-facing changes? New arrow

Followup some doc fixes missed in apache#17888

….0 (apache#17866) ## Which issue does this PR close? - Closes apache#17865. ## What changes are included in this PR? Bump the `substrait` version to `v0.75.0` by bumping `substrait-rs` to `v0.60.0`. This PR was originally dependent on [this PR](apache#17888) to update the versions of some common dependencies, but that PR is now merged in. ## Are these changes tested? There are no tests here, but there is no change to any logic within datafusion. It is simply a bump in a dependency. Technically the public API does change, but as noted in the issue description, there is no change to internal logic because uri / urn from substrait plans are not used. ## Are there any user-facing changes? Yes. Previously substrait plans of spec version `v0.74.0` were accepted, and now `v0.75.0` is accepted. However, this is a backwards compatible change. The only difference is the inclusion of additional urn-based fields in substrait plans. In a later PR, the old uri-based fields will be dropped, which *will* be a breaking change. --------- Co-authored-by: Andrew Lamb <[email protected]>

## Which issue does this PR close? - Related to apache/arrow-rs#7835 - Closes apache#3666 Note while this PR looks massive, a large portion is display updates due to better display of Fields and DataTypes ## Rationale for this change Upgrade to the latest arrow Also, there are several new features in arrow-57 that I want to be able to test including Variant, arrow-avro, and a new parquet metadata reader. ## What changes are included in this PR? 1. Update arrow/parquet 2. Update prost 3. Update substrait 4. Update pbjson 5. Make API changes to avoid deprecated APIs ## Are these changes tested? By CI ## Are there any user-facing changes? New arrow

Followup some doc fixes missed in apache#17888

….0 (apache#17866) ## Which issue does this PR close? - Closes apache#17865. ## What changes are included in this PR? Bump the `substrait` version to `v0.75.0` by bumping `substrait-rs` to `v0.60.0`. This PR was originally dependent on [this PR](apache#17888) to update the versions of some common dependencies, but that PR is now merged in. ## Are these changes tested? There are no tests here, but there is no change to any logic within datafusion. It is simply a bump in a dependency. Technically the public API does change, but as noted in the issue description, there is no change to internal logic because uri / urn from substrait plans are not used. ## Are there any user-facing changes? Yes. Previously substrait plans of spec version `v0.74.0` were accepted, and now `v0.75.0` is accepted. However, this is a backwards compatible change. The only difference is the inclusion of additional urn-based fields in substrait plans. In a later PR, the old uri-based fields will be dropped, which *will* be a breaking change. --------- Co-authored-by: Andrew Lamb <[email protected]>

github-actions bot added the common Related to common crate label Oct 2, 2025

alamb mentioned this pull request Oct 2, 2025

Test thrift-remodel branch with DataFusion apache/arrow-rs#8513

Closed

github-actions bot added substrait Changes to the substrait crate proto Related to proto crate labels Oct 2, 2025

alamb mentioned this pull request Oct 2, 2025

DataType parsing no longer works correctly for old formatted timestamps apache/arrow-rs#8539

Closed

alamb force-pushed the alamb/upgrade_arrow_57 branch from ed43cc0 to ee2de0c Compare October 2, 2025 17:16

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Oct 2, 2025

alamb commented Oct 2, 2025

View reviewed changes

github-actions bot added the core Core DataFusion crate label Oct 2, 2025

alamb force-pushed the alamb/upgrade_arrow_57 branch from 9d06200 to 1b7b559 Compare October 2, 2025 18:56

github-actions bot added the logical-expr Logical plan and expressions label Oct 2, 2025

This was referenced Oct 3, 2025

Refactor: Update enforce_sorting tests to use insta snapshots for easier updates #17900

Merged

Refactor: Split test_window_partial_constant_and_set_monotonicity from enforce_sorting #17903

Closed

alamb force-pushed the alamb/upgrade_arrow_57 branch from 8ecbbed to d3b328b Compare October 3, 2025 15:48

alamb mentioned this pull request Oct 3, 2025

chore(deps): bump substrait from 0.58.0 to 0.60.1 #17841

Closed

alamb force-pushed the alamb/upgrade_arrow_57 branch from f61623e to 9f6a390 Compare October 3, 2025 16:04

alamb mentioned this pull request Oct 3, 2025

Use custom thrift parser for parquet metadata (phase 1 of Thrift remodel) apache/arrow-rs#8530

Merged

github-actions bot added sql SQL Planner physical-expr Changes to the physical-expr crates optimizer Optimizer rules functions Changes to functions implementation physical-plan Changes to the physical-plan crate labels Oct 3, 2025

alamb force-pushed the alamb/upgrade_arrow_57 branch from d5bd26e to 7709acc Compare October 3, 2025 20:26

alamb commented Oct 3, 2025

View reviewed changes

alamb force-pushed the alamb/upgrade_arrow_57 branch from 7709acc to 5e1ea80 Compare October 3, 2025 21:11

alamb mentioned this pull request Oct 4, 2025

[EPIC] Support VARIANT type for unstructured data #16116

Open

4 tasks

alamb mentioned this pull request Oct 4, 2025

[TESTING] Test thrift-remodel branch with datafusion #17916

Closed

alamb changed the title ~~[WIP] Upgrade to arrow/parquet 57.0.0~~ Upgrade to arrow/parquet 57.0.0 Oct 23, 2025

alamb marked this pull request as ready for review October 23, 2025 16:39

Fix compilation

17e7932

paleolimbot mentioned this pull request Oct 23, 2025

chore: Upgrade Datafusion (v50) and Arrow (v56) dependencies apache/sedona-db#237

Merged

getChan approved these changes Oct 24, 2025

View reviewed changes

Jefffrey approved these changes Oct 24, 2025

View reviewed changes

alamb added 4 commits October 24, 2025 15:59

Merge remote-tracking branch 'apache/main' into alamb/upgrade_arrow_57

0a0f53d

Add new syntax example for arrow_cast

1e8ddfa

Update doc examples for arrow_cast

f9606d8

Clarify comments

3a90c6e

alamb changed the title ~~Upgrade to arrow/parquet 57.0.0~~ Upgrade DataFusion to arrow/parquet 57.0.0 Oct 24, 2025

alamb self-assigned this Oct 24, 2025

Merge remote-tracking branch 'apache/main' into alamb/upgrade_arrow_57

31d9684

Merge branch 'main' into alamb/upgrade_arrow_57

552b24f

alamb added this pull request to the merge queue Oct 27, 2025

Merged via the queue into apache:main with commit c6ad17c Oct 27, 2025
36 checks passed

Jefffrey mentioned this pull request Oct 28, 2025

minor: doc fixes for timestamp output format #18315

Merged

github-merge-queue bot pushed a commit that referenced this pull request Oct 28, 2025

minor: doc fixes for timestamp output format (#18315)

8de8621

Followup some doc fixes missed in #17888

tobixdev pushed a commit to tobixdev/datafusion that referenced this pull request Nov 2, 2025

minor: doc fixes for timestamp output format (apache#18315)

7df1546

Followup some doc fixes missed in apache#17888

codetyri0n pushed a commit to codetyri0n/datafusion that referenced this pull request Nov 11, 2025

minor: doc fixes for timestamp output format (apache#18315)

b2db7ab

Followup some doc fixes missed in apache#17888

		// Can disable the cache even with filter pushdown by setting the size to 0. In this case we
		// expect the inner records are reported but no records are read from the cache

Upgrade DataFusion to arrow/parquet 57.0.0 #17888

Upgrade DataFusion to arrow/parquet 57.0.0 #17888

Uh oh!

Conversation

alamb commented Oct 2, 2025 • edited by Jefffrey Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Oct 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Oct 3, 2025

Uh oh!

alamb commented Oct 4, 2025

Uh oh!

alamb commented Oct 23, 2025

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Oct 26, 2025

Uh oh!

alamb commented Oct 27, 2025

Uh oh!

alamb commented Oct 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alamb commented Oct 2, 2025 •

edited by Jefffrey

Loading