Modifying the retention period is causing two schedulers for retention cleanup

[gurjotkaur20]

I want to modify an already set retention period. When I set retention period again, the server schedules retention cleanup as per both the periods.
Ideally it should be set once per stream.

Steps performed

1. The retention period is already set to 1 day in .stream.json "retention": [ { "description": "delete after 1 day", "action": "delete", "duration": "1d" } ]
2. Start the server, scheduler is initialized with this retention period.
3. Set the retention to 3 days. Again scheduler is initialized with 3 days time period.
4. When the retention cleanup happens, it does cleanup for both periods i.e. 1 day and 3 days.

Here are the logs:

[2024-01-26T01:44:22Z INFO  parseable::storage::retention] loading retention for all streams
[2024-01-26T01:44:22Z INFO  parseable::storage::retention] Setting up schedular for backend
[2024-01-26T01:44:22Z INFO  parseable::storage::retention] Scheduler is initialized
[2024-01-26T01:44:22Z INFO  parseable::analytics] Setting up schedular for anonymous user analytics
[2024-01-26T01:44:22Z WARN  parseable] Max priority cannot be set for sync thread. Make sure that user/program is allowed to set thread priority.
[2024-01-26T01:44:22Z INFO  actix_server::builder] starting 8 workers
[2024-01-26T01:44:22Z INFO  actix_server::server] Actix runtime found; starting in Actix runtime
[2024-01-26T01:45:03Z INFO  parseable::storage::retention] Setting up schedular for backend
[2024-01-26T01:45:03Z INFO  parseable::storage::retention] Scheduler is initialized
[2024-01-26T01:45:03Z INFO  actix_web::middleware::logger] 127.0.0.1 "PUT /api/v1/logstream/backend/retention HTTP/1.1" 200 50 "-" "PostmanRuntime/7.29.4" 0.001174
[2024-01-26T01:45:17Z INFO  actix_web::middleware::logger] 127.0.0.1 "GET /api/v1/logstream/backend/retention HTTP/1.1" 200 74 "-" "PostmanRuntime/7.29.4" 0.000451
[2024-01-26T01:46:02Z INFO  parseable::storage::retention::action] running retention task - delete
[2024-01-26T01:46:03Z INFO  parseable::storage::retention::action] running retention task - delete

[nitisht]

Thanks for reporting @gurjotkaur20 .

[gurjotkaur20]

@nitisht To fix this issue, I have two approaches in my mind.

1. We can keep a termination flag(false initially) per stream which will be used to exit this loop.
   

   parseable/server/src/storage/retention.rs

   Line 83 in 90a44d6

   rt.block_on(async move {

   
   Whenever any retention period is set for a stream, a function will check in the hashmap if the flag is false, will be set to true and loop will exit. Then do the init scheduler with new retention period.
2. It requires re-design of the init scheduler method. Instead of scheduling a separate scheduler for each stream, just schedule a single function which will load the retention period of all streams, calls cleanup for each stream in a separate thread. In this case, only single event will be scheduled unlike the current implementation.

What do you suggest?

[nitisht]

It requires re-design of the init scheduler method. Instead of scheduling a separate scheduler for each stream, just schedule a single function which will load the retention period of all streams, calls cleanup for each stream in a separate thread. In this case, only single event will be scheduled unlike the current implementation.

IMHO this is a cleaner approach. However we'll need to ensure the scheduler starts independent workers to actually delete stuff.