Skip to content

Conversation

@chinmay-bhat
Copy link
Contributor

@chinmay-bhat chinmay-bhat commented May 30, 2024

Draft PR. Waiting on incremental append scan #533 to merge.

  • Integration test (test output has both Insert and Delete operations)

I created a new class ChangelogScanTask(ScanTask) to store change_ordinal and other parameters.
The plan_files() in the base class BaseIncrementalScan returns a Iterable[FileScanTask].
pyarrow.project_table() also expects tasks as Iterable[FileScanTask].

TODO:

  • in BaseIncrementalScan, change return type from Iterable[FileScanTask] to Iterable[ScanTask] to allow different scan tasks.
  • [needs discussion] update pyarrow.project_table() to be @singledispatch to allow different behaviours of project_table() depending on the ScanTask type.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant