Custom scripts in sub packages should not be cached

[kkthxbye-code]

NetBox version

v3.4.1

Python version

3.10

Steps to Reproduce

Create the structure described in the following issue:

#10258

Change the Script class name or the variable name of any ScriptVariable.

Expected Behavior

The new code should be dynamically reloaded.

Observed Behavior

Code is not reloaded.

We have had discussions about this at the last maintainers meeting. Dynamically loading code in python is not a pleasent experience, and as such there is a bunch of pitfalls. Currently we delete the module entry in sys.modules when loading the script list, this prevent caching. This doesn't work for nested packages, which this issue seeks to address.

In addition to this, there's other issues that might be also be addressed:

• The way we currently load scripts is to load any module/package in the SCRIPTS_ROOT folder in a way that makes the module path be in the root path. Essentially this means that if we load a file called ipam.py in the scripts folder, it will clash with the already existing ipam module.
• load_module is to be deprecated in python 3.12, so we need to refactor the script/report loading code at some point soon regardless.

I'll be looking into solutions to these problems, but if anyone has experience with dynamically loading python code in a long running python process, please feel free to give input.

For reference, the loading code is here:

netbox/netbox/extras/scripts.py

Lines 513 to 543 in 3675ad2

	def get_scripts(use_names=False):
	"""
	Return a dict of dicts mapping all scripts to their modules. Set use_names to True to use each module's human-
	defined name in place of the actual module name.
	"""
	scripts = {}
	# Iterate through all modules within the scripts path. These are the user-created files in which reports are
	# defined.
	for importer, module_name, _ in pkgutil.iter_modules([settings.SCRIPTS_ROOT]):
	# Use a lock as removing and loading modules is not thread safe
	with lock:
	# Remove cached module to ensure consistency with filesystem
	if module_name in sys.modules:
	del sys.modules[module_name]
	module = importer.find_module(module_name).load_module(module_name)
	if use_names and hasattr(module, 'name'):
	module_name = module.name
	module_scripts = {}
	script_order = getattr(module, "script_order", ())
	ordered_scripts = [cls for cls in script_order if is_script(cls)]
	unordered_scripts = [cls for _, cls in inspect.getmembers(module, is_script) if cls not in script_order]
	for cls in [*ordered_scripts, *unordered_scripts]:
	# For scripts in submodules use the full import path w/o the root module as the name
	script_name = cls.full_name.split(".", maxsplit=1)[1]
	module_scripts[script_name] = cls
	if module_scripts:
	scripts[module_name] = module_scripts
	return scripts