Allow user to create their own DJANGO-RQ queues

[dgarros]

Environment

• Python version: 3.7.7
• NetBox version: 2.8.8

Proposed Functionality

Allow users to create their own DJANGO-RQ queues
Currently NetBox creates 2 queues by default: default and check_releases.
As more features and plugins are leveraging the RQ queues system to execute asynchronous tasks, it would be useful to be able to create more queues.

Use Case

For the onboarding plugin (or other tasks that need to communicate with the devices), it would be useful to have a queue per region to be able to run specific workers per region that are authorized to connect to the network devices (jumphost).

Database Changes

No change to the database.

External Dependencies

No external dependency.

[jeremystretch]

Allow users to create their own DJANGO-RQ queues

Could you expand on exactly what's being proposed? What is the workflow for users to create new queues?

[dgarros]

@jeremystretch My initial thought was to define new queues in the configuration.py file, in a similar way than what django-rq is doing.
If possible, I think a good solution would be to allow the user to define their own RQ_QUEUES, completely or partially. NetBox could add its own queues on top of what the user provides default, check_releases.

[cpmills1975]

Would it make more sense that if a plugin requires a queue for a specific purpose, it is able to be specified in the plugin's setup.py file or PluginConfig sub-class (sorry - I'm not sufficiently knowledgeable to understand the difference)? That way, the requirement to run a queue is kept contained in the plugin code itself and wouldn't cause a problem if a user imported a plugin and didn't modify their site wide configuration.py file.

[dgarros]

@cpmills1975 in some cases it could make sense but not for all cases.
For example, if the goal is to have different queues to control on which jumphost/server a task is running, it won't be possible for the plugin creator to know in advance how many queues are required. Some users will need 1, other will need dozen or more.

[tb-killa]

I agree with @cpmills1975: Plugins should be able to build and use own rq-queue if needing.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our contributing guide.

[dgarros]

Looks like different use cases could have different requirements for this feature.
The main use cases I have in mind is to allow the users to control where a given task will be executed.

For example for the onboarding plugin, It's likely that not all servers will be able to connect directly to the network devices so the worker needs to run on a Jumphost that can connect to the devices. If the Jumphosts are different per region (US/APAC/EMEA) and no jumphost can reach all network devices, it won't be possible to use the onboarding plugin currently because we can't control where a task is running.
From a plugin developer point of view, I would like to provide an option to define where a given onboarding task should run (onboard, 1.2.3.4 in APAC). In this case APAC would be the name of queue previously created by the user which would map to a group of workers running in the APAC jumphost.

In this scenario, the different queues needs to be statically defined by the users ahead of time, in the configuration file. The plugin (or netbox) will just provide an option to select what queue should be used for a given job.