-
-
Notifications
You must be signed in to change notification settings - Fork 153
Closed
Labels
Description
When query occurs and end time of query is greater than ( current time - upload interval ) then there is need to execute query on local data as well ( data that is yet to be synced ). This is handled by execute_on_cache but the datafusion context used is not aware about localsync process. There might be chances that file is moved after listing happened and it is ignored for the query.
Todo
- Custom table provider which maps local data without any issues caused by sync cycle.
- Greater pruning based on query time, limit and offset
- Combined execution model ( will be added after further improvements )