Allow redefining internally used background queues

[kkthxbye-code]

NetBox version

v3.3.8

Feature type

Change to existing functionality

Proposed functionality

Allow changing the used RQ queue for webhook scheduling, custom scripts and reports to a specific queue.

Use case

Currently script/report and webhook execution uses the same queue ("default"). One RQ worker will only ever run one job at a time:

Each worker will process a single job at a time. Within a worker, there is no concurrent processing going on. If you want to perform jobs concurrently, simply start more workers.

For webhooks we want to only ever run one job at the time, as the order is very important to avoid a webhook referencing a related object that hasn't been sent in a webhook yet.

For scripts/reports the user might not care about the order of executions, but as it stands now if the user runs one rqworker, long running scripts will block both webhooks and other scripts from execution before it is done.

If the user starts multiple rqworkers that will allow multiple scripts to run at a time, but it will also allow multiple webhooks to be sent at a time, which in turn can create situations where a newer webhook is done processing before an older one - imagine three workers all picking up a job at the same time, the webhook will potentially send the HTTP request at roughly the same time, which one goes first depends on CPU scheduling or other external factors.

Allowing webhooks to be configured to use a seperate queue will allow the user to start one worker for webhooks and multiple workers for script/report execution, without risking data consistency issues with webhooks.

Database changes

None

External dependencies

None

[kkthxbye-code]

If accepted I'll implement it. We also need to sync up with netbox-docker as there needs to be changes to the default docker-compose file to add the new queue.

[julianstolp]

If adding the ability to set the queue priority for webhooks, would it be possible to add the ability to set the priority in general at the URL with maybe the flag ?priority=high, ?priority=default, ?priority=low. The usecase would be to prioritize manual triggered scripts to run before the background scripts.

[kkthxbye-code]

@Bierfass - This feature request would be to set a static queue for webhooks, as there in my opinion is no reason to want to concurrently execute webhooks because of the mentioned pitfalls of the ordering not being perserved.

Allowing you to schedule scripts and reports in different queues is a good idea though, but would be another feature request. It would probably either be set in the Meta class of the script or as a form selection when executing the script though. Also note that the low, default and high queues only work as intended if running more than one rqworker as far as I can tell.

[jeremystretch]

Just recapping our discussion from today's maintainers' meeting. This is a great idea and I'd like to be able to configure queues per process type. For example, something like this in configuration.py:

QUEUE_MAPPINGS = {
    'webhooks': 'my-custom-queue',
    'reports': 'high',
}

This would grant users a good amount of flexibility to create and assign their own queues as needed (each of which can be serviced by however many RQ workers is appropriate).

[hagbarddenstore]

Will this be extended to plugins as well, so that they register their keys in that map?