Move leftover data.parquet to tmp with valid prefix on server startup #73
Conversation
| if !dir.parquet_path_exists() { | ||
| continue; | ||
| } | ||
| if let Err(e) = dir.create_temp_dir() { |
There was a problem hiding this comment.
what is the behaviour here if the tmp directory exists already? I mean we need to ensure we don't overwrite, because it is highly probably that the directory exists and has some data that is still not sent to S3.
There was a problem hiding this comment.
create_dir_all recursively builds all the missing component of path. If there is already an existing tmp, it will not overwrite anything inside
nitisht
left a comment
nitisht
left a comment
There was a problem hiding this comment.
This is not working on macOS and mostly windows as well. For platforms where we can't fetch the last modified time, we still need to discard data.parquet if that exists in the ./data directory. Otherwise we'd be adding new data to stale data from when the server crashed
|
seems to be working now. |
2 participants
Fixes #53 .
Description
When server is restarted it should try to sync data on the local directory that is not yet synced to s3. Otherwise the data will get lost when new event arrives.
Make use of last modified system time to generate a prefix and move
data.parquetto tmp directory, It will be synced to s3 on nexts3_syncrun.This PR has: