Improve Performance of Generating Config Contexts

[dstarner]

Environment

• Python version: 3.7
• NetBox version: 2.7.12, but issue exists on <= 2.8.1

Proposed Functionality

Improve the get_config_context serializer method that is observed to have slow performance that builds up as the number of fetched devices at a time increases.

I'm not sure exactly where the bottleneck is, but it does perform a very complex query that may be the cause of the overhead. I performed some profiling awhile back and noticed that the SQL queries used an acceptable amount of request time, but that it was mostly using CPU execution time. I'm not sure if this is still the case across the board, but it may be worthwhile to profile and optimize whatever calls are made under this get_config_context serializer method.

Use Case

Fetching 1000 devices by default takes ~60s. Without config context - using ?excludes=config_context - the response takes ~5 seconds. This large overhead was determined to be from the generation & merging of the config contexts that exist on devices / virtual machines. Reducing this performance overhead would make NetBox's usage much more friendly and acceptable.

Database Changes

This context is regenerated on every request on via this serialize. This then calls this complex query which either in the query, object generation, or serialization takes an enormous amount of time.

We would need to evaluate the performance of this query / method and determine where the bottleneck is occurring, and how to make it faster.

External Dependencies

N/A

---

Note that this is a revised version of #4544

[dstarner]

@lampwins do you have any ideas / advice on how we could make these queries more performant? Sadly this is slightly out of my area of expertise.

[DouglasHeriot]

This affects my use of Netbox (via the netbox-community/ansible_modules inventory plugin) - I can confirm it also takes me about ~60s to query 1000 devices.

If nobody else gets to it first, I might have a go at digging into profiling what's going on here at some point in the next few months.

[tyler-8]

As @lampwins noted in the previous issue:

The reason this is so inefficient today is that this method gets called on each serialized device instance which in turn calls this complex query logic for every device.

[lampwins]

I dove into this a bit over the weekend and I think there are two primary ways we can approach this. The first is coming up with a reasonable way to do the deep merge of context data in psql and the second is doing some sort of annotation on the config context object query to have psql do the mapping for us in bulk.

[kevinreniers]

Thanks to @tyler-8 for pointing me to this issue. I've also noticed that redis performance is significantly impacted by this.

We have a number of API clients regularly calling the /api/dcim/devices endpoint for various limits and at various offsets. We noticed that the Redis server's CPU usage would spike to near 100% with just two or three simultaneous calls. As suggested in this thread, excluding config_contexts from the response alleviates this problem entirely. As soon as we did that, CPU usage dropped back down to below 0.5%.

Might I suggest that this functionality gets disabled by default for now, and that the API docs mention the very significant performance impact of enabling it?

Aside from that, couldn't this problem be solved by regenerating the config_contexts in a background task when an action happens that requires it to change, rather than generating it on-demand for every API call?

[tyler-8]

Might I suggest that this functionality gets disabled by default for now, and that the API docs mention the very significant performance impact of enabling it?

Might be something to discuss in the Slack channel or Google Group before opening an issue to improve docs.

Aside from that, couldn't this problem be solved by regenerating the config_contexts in a background task when an action happens that requires it to change, rather than generating it on-demand for every API call?

That was the initial topic that led to this issue. #4544 - the decision was that the logic to generate config contexts needs to be "fixed" first as the way it's currently written invokes multiple DB queries PER device in a query of N-devices, rather than a handful of queries.

[dstarner]

What would be the best way to remove these queries? I don't mind working on it a bit, I just don't want to see this issue fall by the wayside as its pretty important to my team to fix.

[danielestevez]

Maybe making this excludes=config_context the default option could work as a quickfix for a minor version?
This is quite a blocker to upgrade to a newer version of Netbox since there's no way we can control how third party tools use the Netbox API

[zacho112]

Quick fix to implement the exclude="config_context" across our codebase:

Rewrites all filter to include the exclude (or append it), and the all() method to a filter() aswell.

import pynetbox
org_filter = pynetbox.core.endpoint.Endpoint.filter


def new_filter(self, *args, **kwargs):
    excludes = kwargs.get("exclude", [])
    if isinstance(excludes, str):
        excludes = [excludes]

    if "config_context" not in excludes:
        excludes.append("config_context")

    kwargs["exclude"] = excludes

    return org_filter(self, *args, **kwargs)


pynetbox.core.endpoint.Endpoint.filter = new_filter
pynetbox.core.endpoint.Endpoint.all = new_filter


nb = pynetbox.api(
    netbox_url,
    token=netbox_token,
    ssl_verify=False
)

start = time.time()
devices = nb.dcim.devices.all()
print(time.time() - start)
print(len(devices))

[jeremystretch]

Tagging this as under review until a specific implementation has been identified.