Database Locking Backend

[msopko81]

I believe the default locking backend should be a database backend using django_cron_cronjoblog database table.

When the job is first started, a row could be added into the database table with a start_time and code. This can then be used to prevent parallel runs (if needed). When the job finishes, the end_time, is_success, and message can be updated.

This has the benefit of not requiring the django cache backend. It will also prevent issues when not using the django cache since the default cache backend is local memory and nothing is being used to prevent parallel runs.

[agronick]

Cache is atomic. Database is not. I vote for staying with cache so I can be sure there will never be two locks that were created at the same time.

[kukosk]

@agronick That could possibly happen also with the current implementation of cache, if .set() would be called from another thread between .get() and .set() calls ... And with transactions, this could be made atomic also with a db backend :)

[agronick]

@kukosk Thats Django has the get_or_set() cache method to protect against that race condition. I suppose it could be done with the database if you set up integrity checks in the table creation.

[msopko81]

Since the cache can be backed by a database, you cannot say cache is atomic and database is not. Django also has get_or_create method for the database. get_or_create returns a flag if the row was created. If the row was created, then it has the lock to run, otherwise, return saying there is already a lock. I would prefer to use select_for_update instead of get_or_create

Using get_or_set would not work because if 1 thread sets the lock, the second thread using get_or_set will have the data returned and not have an error thrown.

My main issue is using this on Windows and the task scheduler not being as good as the unix cron. I have the task triggered every 5 mins. At 3:05 and 3:10 the task was not triggered. They both ran at 3:12.

Using the built in file locking is not removing the files all of the time so the jobs stay locked. I tried to use Django's DB cache and it is not removing the rows (no idea why this is happening). I tried using a pull request for kernal based locking and it is not working on windows. I used Django's file based lock and it mostly worked, except for what I mentioned. Because the jobs were running twice, 1 job created the lock file and the other job failed getting/clearing the lock. This is a false positive. The job didn't even need to run, but threw an error.

I also tried REDIS backend, but had issues with REDIS stopping. We also need to do weekly reboots on windows otherwise something eats up all of the memory (gotta love windows). I was about to write a DB route to use a sqlite DB for caching and implementing a DB timeout to wait for writing to the DB, but decided to use https://github.com/grantjenks/python-diskcache instead which uses sqlite DB and file based caching.

I also had to write my own run_crons management command because the pull request with multi-threading was trying to do too many jobs at the same time. I had to limit it to 4 at a time (which is working great at the moment).

[laixintao]

Any plan on that?

We have a django without cache, but 4 uwsgi servers. It could be useful if we make this happen.

[msopko81]

I thought about setting up a pull request, but I do this and it is accepted, then I would also remove everything with the cache and would change a lot of the flow and design. At that point, it would either be to go down this route, or fork it off as a new project entirely.

[laixintao]

Why remove cache? Why don't you just keep both and let users make the choice to use db or cache?

[msopko81]

My thought was, if you are going to do the DB backend to keep track of everything, then the cache just becomes overhead. Why look at the cache when we already have the DB keeping track of everything. You are also adding an additional level of complexity with the cache that would no longer be needed since you would then need to check the cache and then the DB.

Maybe I can take another look and do a cache check first, if a cache is provided.

[laixintao]

@msopko81 , I agree with you, my only concern is that if you remove cache it will break others' code.

[msopko81]

I wouldn't worry about that since the code would no longer look at the other variables in the settings. They would be there, but just not used. Using the DB, the cache would just be a faster way to see if there is a lock, but the DB would always have to be the master of checking for the locks.