added setting to run all cronjobs in parallel

[FrankMeyer]

DJANGO_CRON_MULTITHREADED, default is False
Each job is run in a single thread, this does not
affect the check for temporal overlaps of the same cronjob

[FrankMeyer]

This is usefull in case of multiple tasks sheduled at the same time.
Then they are started all at the exact sheduled time.
Multithreading is also useful in case you have many different tasks that should run every minute or so.
Those will then not be sheduled one after the other anymore.

[tab-cmd]

Allowing parallels runs

By deafult parallels runs are not allowed (for security reasons). However if you want enable them just add:

ALLOW_PARALLEL_RUNS = True
in your CronJob class.

Note

Note this requires a caching framework to be installed, as per https://docs.djangoproject.com/en/dev/topics/cache/

If you wish to override which cache is used, put this in your settings file:

DJANGO_CRON_CACHE = 'cron_cache'

[FrankMeyer]

Hello Sci-Tab,

you have misunderstood me, I don't want to run one CronJob in parallel, I want to start all different CronJobs that are enabled in parallel. I have multiple different Jobs to do and the starting times will not be exact, because the starting time of a job requires the jobs before to have ended (see runcrons.py). That is pretty bad, so I have added that option to start each different CronJob in a differend thread. Disallowing parallel runs of one CronJob still works.

greetings, Frank

[lauritzen]

Seems like a good and simple solution. @sci-tab any chance it can be merged in?

[msopko81]

I would suggest having a variable to limit the number of threads running at a time. I tried this solution an had issues since it was trying to run all 60 jobs/threads at the same time.

[laixintao]

@msopko81 What's your problem exactly? Maybe we can add another variable to settings to set the max threading numbers.

[msopko81]

Well, if you look at #114, I explained the issues I had. The main issue I had with this was all of the jobs running at the same time with an sqlite DB for cache and all 60 jobs trying to run at the same time cause the jobs to wait a long time trying to get sqlite write access. I had since switched back to REDIS (less errors occurring) and it is no so much of an issue, but other may have similar issues I had and would like to limit the number of concurrent threads running at the same time.

[lauritzen]

I can't see that @msopko81 problem should be a blocker. For those special cases just keep the default config. Any update on when this will me merged in?

[bradyemerson]

I might suggest that instead of using a true/false flag, have a INT variable for the number of threads with a default of 1. You could theoretically avoid the fork in the code, and you wouldn't need two settings to capture the behaviors discussed in the comments (threads Y/N and num of threads).