Skip to content

Move to Datafusion 26 #438

@trueleo

Description

@trueleo

Description

Discussion and plans for moving to Datafusion 26.
Datafusion 26 comes with lot of exciting features and thus changes to Parseable.

We currently update arrow and datafusion simultaneously for simplicity.

Arrow 40

  • Schema field has changed from Vec to Fields ( Arc<[Arc<Field>]> )
  • JSON -> RecordBatch flow is now faster and supports direct serialization from serde compatible structs

Datafusion 26
Better support for timestamp based queries. SELECT now() + '1 month' FROM my_table;
Better support for list and struct datatypes. Some insights require to see if schema can be evolved in list of strut case

Stage Data Query:
Since Datafusion supports querying arrows files as well, we can target enabling staging data query.

Moved to separate issues

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions