Allow plugin to create their own django-rq queues

[maximumG]

NetBox version

v2.11.7

Feature type

New functionality

Proposed functionality

I'd like to propose the ability for netbox plugin to define some dedicated django-rq queue(s). This new feature has already been discussed in #4908 but was missing a owner back in time.

The idea would be to store queue names in a new attribute inside the PluginConfig class. This attributes would be a list in order to defined several queue to create by django-rq.

In order to avoid queue name collision, the actual queue names will be prepended by the plugin name.

As an exemple the following config will lead to 3 queues being created using django-rq:

• my_super_plugin.high
• my_super_plugin.medium
• my_super_plugin.low

MyPluginConfig(PluginConfig):
    name = "my_super_plugin"
    queues = [
        "low"
        "medium"
        "high"
    ]

The creation of the queues will need to override django settings (RQ_QUEUES) and can be done in the ready method of the PluginConfig class.

We can also override django RQ_QUEUES settings directly in the settings.py, during init of all plugins.

If you are interested in such feature I can take care of a PR.

Use case

All netbox plugins have the possibility to use the default queue to do some background/housekeeping tasks. If these tasks are long-running tasks they will block all other netbox-native background jobs, including netbox-reports & netbox-script.

Database changes

No response

External dependencies

No response

[maximumG]

I already added this feature in my own netbox fork along with some new unittest functions to ensure the implementation is working as expected

If we agree on this I can move on and update the netbox documentation to add some notes regarding dedicated plugin queues.

[jeremystretch]

Thanks for this!

All netbox plugins have the possibility to use the default queue to do some background/housekeeping tasks. If these tasks are long-running tasks they will block all other netbox-native background jobs, including netbox-reports & netbox-script.

Creating multiple queues is just half the battle though, right? They also need to be ranked according to priority so that rqworker services each queue respectively. We need a way to order queues among both NetBox core and one or more plugins in a predictable manner. For example, suppose two different plugins each introduce their own high- and low-priority queues. We don't want to end up ordering them as:

[default, plugin1-high, plugin1-low, plugin2-high, plugin2-low]

Without such a mechanism in place, plugins would be relying on administrators to manually reconfigure their rqworker invocations accordingly, which is sure to lead to frustration. I'm open to suggestions.

NetBox also currently has a check_releases queue that is used solely for checking for new releases. I don't recall the motivation for this design decision initially, but it's probably best moved to a scheduled task within the default queue using rq-scheduler anyway. (In fact, this may become necessary if we decided to remove caching support as discussed under #6639.)

[maximumG]

Creating multiple queues is just half the battle though, right? They also need to be ranked according to priority so that rqworker services each queue respectively.

You are absolutely right, I forgot to mention half of the proposed solution...sorry for this 😞

We need a way to order queues among both NetBox core and one or more plugins in a predictable manner.

From my point of view, queue administration should be tackled at the sysadmin level. What I had in mind in the first place is to have:

• per default, an rqworker instance listening to queue default and check_release, to handle netbox core jobs (already the case now)
• a new rqworker instance configured per plugin. This new instance must be configured to listen only to the plugin's queues

The second point implies that the plugin maintainer inform potential user that they must create a dedicated rqworker instance listening to the plugin queues (container / systemd unit). No matter the underlying implementation to spawn new rqworker it is really simple nowadays to add an additional container/pod (cloud) or systemd unit (on-premise) to handle each plugin background workload.

Even if creating new rqworker instance is a manual operation done by sysadmin, I see the following benefits:

• Each plugin background workload is isolated and does not impact other plugin or netbox core workload
• Netbox source code is "kept clean" and focused on IRM only, as we do not add some code to handle queueing system
• We keep it relatively simple by separating concerns of Netbox code and system implementation details

[jeremystretch]

Maybe there's a middle ground, where NetBox provides its own high- and low-priority queues (in addition to the default queue) for consumption by plugins. NetBox's rqworker would service all three of these accordingly by default. Then I'd feel more comfortable putting the burden on the user to service any additional queues needed by plugins.

[maximumG]

I will try to sum up our two ideas to define a proposal target design.

What Netbox would provides out of the box

Netbox would define 3 queues : default, high-priority and low-priority. The default queue is already present and dedicated for netbox internal-use (reports and scripts jobs). The high-priority and low-priority queues are provided out of the box to plugin developer if they need background job and do not want to implement their own queues.

Netbox would provides a single rq-worker (systemd/container) which listen to the 3 queues above. As stated by the RQ documentation, the queue order is important here and should stay as default, high-priority and low-priority. This way we ensure default job > high-priority jobs > low-priority jobs

What Netbox would allow for plugin developers

Plugin developers have the ability to define their own dedicated queue(s) in the plugin's config. This implies a new rq-worker instance must be created and configured to listen to the plugin dedicated queue(s). In such case, end-user of the plugin must ensure this rq-worker is installed and configured properly according to any plugin install documentation.

[jeremystretch]

I like that! Just one tweak:

the queue order is important here and should stay as default, high-priority and low-priority.

I would arrange them high > default > low. NetBox would continue to queue webhooks, reports, etc. in the default queue.

[jeremystretch]

I pinged the Docker folks for their take on this as well. Per @tobiasge in the #netbox-docker channel on the NetDev Slack:

In the docker-compose.yaml we have defined a container to run the default rqworker. See this configuration. To define additional workers one would need to add a new worker instance to the docker-compose.override.yml file, which is also explained in our documentation.
From that point of view, I think it is easy to add new workers for plugins. So no objections from me.