Exception during migration from 2.9.11 to 2.10.2 when customfieldvalue references non-existent object

[candlerb]

Environment

• Python version: 3.6.9
• NetBox version: 2.9.11 -> 2.10.2

Steps to Reproduce

1. Started with Netbox 2.9.11 where there is some custom field data (*)
2. git pull and ./upgrade.sh

(*) specifics below

Expected Behavior

Data to be migrated

Observed Behavior

  Applying extras.0051_migrate_customfields...Traceback (most recent call last):
  File "netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 245, in handle
    fake_initial=fake_initial,
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/netbox/netbox/extras/migrations/0051_migrate_customfields.py", line 69, in migrate_customfieldvalues
    cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value)
TypeError: 'NoneType' object is not subscriptable

Analysis

After adding some debugging:

--- a/netbox/extras/migrations/0051_migrate_customfields.py
+++ b/netbox/extras/migrations/0051_migrate_customfields.py
@@ -66,6 +66,13 @@ def migrate_customfieldvalues(apps, schema_editor):
         # TODO: This can be done more efficiently once .update() is supported for JSON fields
         cf_data = model.objects.filter(pk=cfv.obj_id).values('custom_field_data').first()
         try:
+            print("***")
+            print("cfv =", cfv)
+            print("cfv.obj_id =", cfv.obj_id)
+            print("cfv.field =", cfv.field)
+            print("cfv.serialized_value =", cfv.serialized_value)
+            print("cf_data =", cf_data)
+            print("cf_data.get('custom_field_data') =", cf_data.get('custom_field_data'))
             cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value)
         except ValueError as e:
             print(f'{cfv.field.name} ({cfv.field.type}): {cfv.serialized_value} ({cfv.pk})')

Result:

...
***
cfv = CustomFieldValue object (241)
cfv.obj_id = 125
cfv.field = CustomField object (9)
cfv.serialized_value = 40
cf_data = {'custom_field_data': {}}
cf_data.get('custom_field_data') = {}
***
cfv = CustomFieldValue object (20)
cfv.obj_id = 133
cfv.field = CustomField object (1)
cfv.serialized_value = 2
cf_data = None
Traceback (most recent call last):
...
  File "/opt/netbox/netbox/extras/migrations/0051_migrate_customfields.py", line 75, in migrate_customfieldvalues
    print("cf_data.get('custom_field_data') =", cf_data.get('custom_field_data'))
AttributeError: 'NoneType' object has no attribute 'get'

So the problem is that cf_data is None. In turn:

netbox=# select * from extras_customfieldvalue where id=20;
 id | obj_id | serialized_value | field_id | obj_type_id
----+--------+------------------+----------+-------------
 20 |    133 | 2                |        1 |          23
(1 row)

netbox=# select * from django_content_type where id=23;
 id | app_label | model
----+-----------+--------
 23 | dcim      | device
(1 row)

netbox=# select id,name from dcim_device where id=133;
 id | name
----+------
(0 rows)

The problem occurs when there is some left-over customfieldvalue which references a non-existent object.

Proposed fix

Print a warning and skip:

--- a/netbox/extras/migrations/0051_migrate_customfields.py
+++ b/netbox/extras/migrations/0051_migrate_customfields.py
@@ -65,6 +65,9 @@ def migrate_customfieldvalues(apps, schema_editor):
         # Read and update custom field value for each instance
         # TODO: This can be done more efficiently once .update() is supported for JSON fields
         cf_data = model.objects.filter(pk=cfv.obj_id).values('custom_field_data').first()
+        if cf_data is None:
+            print(f'{cfv.field.name} ({cfv.field.type}): {cfv.serialized_value} ({cfv.pk}): references non-existent {model.__name__} {cfv.obj_id}')
+            continue
         try:
             cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value)
         except ValueError as e:

[candlerb]

Update: I have successfully updated the system with this fix.

I have also traced where the bad data came from. It originated way back (like 2017), in the days before Netbox had VirtualMachines, and I had modelled VMs as Devices.

When Netbox gained VirtualMachines I moved them from dcim_device to virtualization_virtualmachine (keeping the same id), but I forgot to update the related extras_customfieldvalue rows - I should have either updated the object_type_id, or deleted the row entirely. Hence these dangling values were still pointing at "devices" which no longer exist.

[DanSheps]

If you want to submit a PR for this that would be swell.