`reindex` manage command behaves unexpectedly

[peteeckel]

Deployment Type

Self-hosted

NetBox Version

v4.0.6

Python Version

3.11

Steps to Reproduce

I was made aware of this by a question posted to the Slack channel.

A user asked whether the reindex management command deletes all search indexes except the ones for the application given as a parameter if reindex is used with an application name, for example

/opt/netbox/netbox/manage.py reindex netbox_dns

It turns out that this is in fact the case. The code for the management command always deletes all search indexes even if it was called with an application/plugin name as its parameters, but recreates only the ones for the given application or plugin.

First, I indexed the whole NetBox DNS inventory for all models:

/opt/netbox/netbox/manage.py reindex
Reindexing 82 models.
Clearing cached values... 467 entries deleted.
Indexing models
  netbox_dns.record... 365 entries cached.
  netbox_dns.view... 4 entries cached.
  netbox_dns.nameserver... 7 entries cached.
...
  wireless.wirelesslan... No objects found.
  wireless.wirelesslangroup... No objects found.
  wireless.wirelesslink... No objects found.
Completed. Total entries: 650

The followinig SQL query returns all application/model name pairs that are in the index:

(netbox) [root@dns netbox]# /opt/netbox/netbox/manage.py dbshell
psql (15.5)
Type "help" for help.

netbox=# SELECT DISTINCT (app_label, model) FROM django_content_type c INNER JOIN extras_cachedvalue v ON c.id=v.object_type_id;
           row           
-------------------------
 (core,datafile)
 (core,datasource)
 (dcim,device)
 (dcim,devicerole)
 (dcim,devicetype)
 (dcim,interface)
 (dcim,manufacturer)
 (dcim,site)
 (dcim,virtualchassis)
 (extras,customfield)
 (extras,webhook)
 (ipam,ipaddress)
 (ipam,prefix)
 (ipam,vlan)
 (ipam,vlangroup)
 (ipam,vrf)
 (netbox_dns,contact)
 (netbox_dns,nameserver)
 (netbox_dns,record)
 (netbox_dns,registrar)
 (netbox_dns,view)
 (netbox_dns,zone)
 (tenancy,tenant)
 (tenancy,tenantgroup)
 (vpn,ikepolicy)
 (vpn,ikeproposal)
 (vpn,ipsecpolicy)
 (vpn,ipsecprofile)
 (vpn,ipsecproposal)
(29 rows)

Now, reindex only the search indices for a given application:

(netbox) [root@dns netbox]# /opt/netbox/netbox/manage.py reindex netbox_dns
Reindexing 6 models.
Clearing cached values... 650 entries deleted.
Indexing models
  netbox_dns.record... 365 entries cached.
  netbox_dns.view... 4 entries cached.
  netbox_dns.nameserver... 7 entries cached.
  netbox_dns.zone... 78 entries cached.
  netbox_dns.contact... 10 entries cached.
  netbox_dns.registrar... 3 entries cached.
Completed. Total entries: 467

The same SQL query now shows that the indices for all other applications are missing:

netbox=# SELECT DISTINCT (app_label, model) FROM django_content_type c INNER JOIN extras_cachedvalue v ON c.id=v.object_type_id;
           row           
-------------------------
 (netbox_dns,contact)
 (netbox_dns,nameserver)
 (netbox_dns,record)
 (netbox_dns,registrar)
 (netbox_dns,view)
 (netbox_dns,zone)
(6 rows)

Deleting all other index can only be avoided by also specifying --lazy, but that option prevents the indices for a specific application or model to be recreated (i.e., deleted and then created again). There is currently no way to achieve this.

Expected Behavior

Only the search indexes for the specified applications/models are deleted, not all of them, and certainly not silently.

Observed Behavior

All indexes are deleted and only those for the specified application/model are (re-)created.

[arthanson]

This code looks like it clears all at start

        # Clear all cached values for the specified models (if not being lazy)
        if not kwargs['lazy']:
            self.stdout.write('Clearing cached values... ', ending='')
            self.stdout.flush()
            deleted_count = search_backend.clear()
            self.stdout.write(f'{deleted_count} entries deleted.')

[peteeckel]

This code looks like it clears all at start

exactly. Which isn't a problem if you don't specify a list of applications/models, but if you do it does unexpected thinks.

You can assign it to me if that's OK for you.

[arthanson]

@peteeckel assigning to you.