Skip to content

Improve query execution #87

@trueleo

Description

@trueleo

When query occurs and end time of query is greater than ( current time - upload interval ) then there is need to execute query on local data as well ( data that is yet to be synced ). This is handled by execute_on_cache but the datafusion context used is not aware about localsync process. There might be chances that file is moved after listing happened and it is ignored for the query.

Todo

  • Custom table provider which maps local data without any issues caused by sync cycle.
  • Greater pruning based on query time, limit and offset
  • Combined execution model ( will be added after further improvements )

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions