Skip to content

Query staging data by reading .arrows files #445

@trueleo

Description

@trueleo

Tracking issue for local query.

Todo:

  • Add .arrows format by implementing FileFormat. Datafusion only supports IPC file format not the streaming format which we use.
  • Add arrow partition streams to table provider
  • While reading compact record batches to save space.
  • Create table provider that joins local and remote query together in right order.
  • ( Optional ) Support projection pushdown.
  • Add atomic file references so that no arrow files are removed during query.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions