User and Group object types missing from permissions in 4.0

[ibuclaw]

Deployment Type

Self-hosted

NetBox Version

v4.0.2

Python Version

3.11

Steps to Reproduce

In the UI:

1. Admin > Permissions
2. Add or Edit a permission
3. Inspect list of selectable Object types.
4. Authentication and Authorization > Group still exists, but it does nothing (Admin > Groups is visible on the navigation bar, but accessing it returns 403 permission denied).
5. Authentication and Authorization > User is gone, and users who used to have that permission can no longer view/edit the Users page.

In Postgresql:

1. Upgrade from 3.7.8 to 4.0.2 and run migrations
2. psql -d netbox
3. \d
4. Observe that auth_user and all other related tables are gone (suspect one of the migrations did it but it's unclear which).
5. Observe that auth_group and auth_group_permissions remains but are empty (it's more clear that this is the migration, no apparent deletion of the table is present).

Expected Behavior

I see that there's a couple migration scripts. I suspect that the DB migration from auth_user to users_user is incomplete but don't understand enough to definitively say so.

The ability to manipulate Users and Groups should be possible by users without superuser powers. This is more obviously broken since 4.0.x.

Observed Behavior

Users with permission to view/change users and groups get 403 permission denied when accessing those pages.

[ibuclaw]

I think this would sort out the oddities on the navigation menu, but would mean users can't see either the Users or Groups admin menu items:

--- a/netbox/netbox/navigation/menu.py
+++ b/netbox/netbox/navigation/menu.py
@@ -372,19 +372,19 @@ ADMIN_MENU = Menu(
                     link=f'users:user_list',
                     link_text=_('Users'),
                     auth_required=True,
-                    permissions=[f'auth.view_user'],
+                    permissions=[f'users.view_user'],
                     buttons=(
                         MenuItemButton(
                             link=f'users:user_add',
                             title='Add',
                             icon_class='mdi mdi-plus-thick',
-                            permissions=[f'auth.add_user']
+                            permissions=[f'users.add_user']
                         ),
                         MenuItemButton(
                             link=f'users:user_import',
                             title='Import',
                             icon_class='mdi mdi-upload',
-                            permissions=[f'auth.add_user']
+                            permissions=[f'users.add_user']
                         )
                     )
                 ),
@@ -392,19 +392,19 @@ ADMIN_MENU = Menu(
                     link=f'users:group_list',
                     link_text=_('Groups'),
                     auth_required=True,
-                    permissions=[f'auth.view_group'],
+                    permissions=[f'users.view_group'],
                     buttons=(
                         MenuItemButton(
                             link=f'users:group_add',
                             title='Add',
                             icon_class='mdi mdi-plus-thick',
-                            permissions=[f'auth.add_group']
+                            permissions=[f'users.add_group']
                         ),
                         MenuItemButton(
                             link=f'users:group_import',
                             title='Import',
                             icon_class='mdi mdi-upload',
-                            permissions=[f'auth.add_group']
+                            permissions=[f'users.add_group']
                         )
                     )
                 ),

[ibuclaw]

This fixes the missing Object types on the permissions page, but the auth_group table is still missing a clean up in the database.

--- a/netbox/users/constants.py
+++ b/netbox/users/constants.py
@@ -3,8 +3,7 @@ from django.db.models import Q
 
 OBJECTPERMISSION_OBJECT_TYPES = Q(
     ~Q(app_label__in=['account', 'admin', 'auth', 'contenttypes', 'sessions', 'taggit', 'users']) |
-    Q(app_label='auth', model__in=['group', 'user']) |
-    Q(app_label='users', model__in=['objectpermission', 'token'])
+    Q(app_label='users', model__in=['group', 'objectpermission', 'token', 'user'])
 )
 
 CONSTRAINT_TOKEN_USER = '$user'

[ibuclaw]

W.R.T why Authentication and Authorization > Groups is still visible on the list of Object types, but Authentication and Authorization > Users is not, the contents of django_content_types seems to explain it.

 id  |       app_label       |           model           
-----+-----------------------+---------------------------
   1 | extras                | webhook
   2 | admin                 | logentry
   3 | auth                  | permission
   4 | auth                  | group
   6 | contenttypes          | contenttype
   7 | sessions              | session
   8 | social_django         | association
   9 | social_django         | code
  10 | social_django         | nonce
  11 | social_django         | usersocialauth
  12 | social_django         | partial
  13 | taggit                | tag
  14 | taggit                | taggeditem
  15 | core                  | datasource
  16 | core                  | datafile
  17 | core                  | autosyncrecord
  18 | core                  | managedfile
  19 | core                  | job
  21 | core                  | configrevision
...
 155 | users                 | group
   5 | users                 | user
 157 | core                  | objecttype

As part of the migration, it looks like auth | user got renamed to users | user, whereas users | group was just created as a new entry, leaving auth | group there to rot.

I note differences between the two migration scripts for users and group ContentTypes, but it is unclear how this affects the contents of django_content_type.

[ibuclaw]

Started from a clean installation

1. Install Netbox v3.7.8 using git method as per https://docs.netbox.dev/en/stable/installation/
2. Add admin permission

---

3. Add admin group with admin permission

---

4. Add admin user with admin group and Staff status (but not Superuser).

---

5. Login as admin user and check that Users and Groups are visible/editable.
   
   

---

Inspect the database to get a grasp of how things connect in v3.7.8

netbox=# select id, username, is_superuser, is_staff from auth_user;
 id | username | is_superuser | is_staff 
----+----------+--------------+----------
  1 | root     | t            | t
  2 | jsmith   | f            | t
(2 rows)

I see two users.

netbox=# select * from auth_group;
 id | name  
----+-------
  1 | admin
(1 row)

I see one group.

netbox=# select * from django_content_type where app_label in ('auth', 'users');
 id  | app_label |      model       
-----+-----------+------------------
   3 | auth      | permission
   4 | auth      | group
   5 | auth      | user
 115 | users     | userconfig
 116 | users     | token
 117 | users     | objectpermission
 118 | users     | netboxgroup
 119 | users     | netboxuser
(8 rows)

I see the group and user permissions will be referenced by the content_type id 4 and 5.

netbox=# select * from auth_permission where content_type_id in (4, 5);
 id |       name       | content_type_id |   codename   
----+------------------+-----------------+--------------
  9 | Can add group    |               4 | add_group
 10 | Can change group |               4 | change_group
 11 | Can delete group |               4 | delete_group
 12 | Can view group   |               4 | view_group
 13 | Can add user     |               5 | add_user
 14 | Can change user  |               5 | change_user
 15 | Can delete user  |               5 | delete_user
 16 | Can view user    |               5 | view_user
(8 rows)

netbox=# select * from users_objectpermission;
 id | name  | description | enabled |         actions          | constraints 
----+-------+-------------+---------+--------------------------+-------------
  1 | admin |             | t       | {view,add,change,delete} | 
(1 row)

I see the permission have just created

netbox=# select * from users_objectpermission_groups;
 id | objectpermission_id | group_id 
----+---------------------+----------
  1 |                   1 |        1
(1 row)

Seems to link the permission to the admin group

netbox=# select * from users_objectpermission_object_types;
 id | objectpermission_id | contenttype_id 
----+---------------------+----------------
  1 |                   1 |              4
  2 |                   1 |              5
(2 rows)

Seems to link the permission to the group (4) and user (5) content types.

Upgrade process instance to latest release

1. Upgrade Netbox to v4.0.2 using git method as per https://docs.netbox.dev/en/stable/installation/upgrading/

Operations to perform:
  Apply all migrations: account, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, vpn, wireless
Running migrations:
  Applying dcim.0186_location_facility... OK
  Applying dcim.0187_alter_device_vc_position... OK
  Applying extras.0108_convert_reports_to_scripts... OK
  Applying extras.0109_script_model... OK
  Applying extras.0110_remove_eventrule_action_parameters... OK
  Applying extras.0111_rename_content_types... OK
  Applying extras.0112_tag_update_object_types... OK
  Applying extras.0113_customfield_rename_object_type... OK
  Applying extras.0114_customfield_add_comments... OK
  Applying extras.0115_convert_dashboard_widgets... OK
  Applying tenancy.0015_contactassignment_rename_content_type... OK
  Applying users.0005_alter_user_table... OK
  Applying users.0006_custom_group_model... OK
  Applying users.0007_objectpermission_update_object_types... OK
  Applying users.0008_flip_objectpermission_assignments... OK
  Applying vpn.0005_rename_indexes... OK
Checking for missing cable paths (python3 netbox/manage.py trace_paths --no-input)...
Found no missing console port paths; skipping
Found no missing console server port paths; skipping
Found no missing interface paths; skipping
Found no missing power feed paths; skipping
Found no missing power outlet paths; skipping
Found no missing power port paths; skipping
Finished.

---

2. Refresh page after service restart.

---

3. Logout/Login as superuser and inspect the admin permissions.

---

4. Edit the admin permission and look for the new Users and Groups object types.
   
   

---

Inspect the database again to get a grasp of how things connect in v4.0.2

netbox=# select id, username, is_superuser, is_staff from auth_user;
ERROR:  relation "auth_user" does not exist
LINE 1: select id, username, is_superuser, is_staff from auth_user;
                                                         ^
netbox=# select id, username, is_superuser, is_staff from users_user;
 id | username | is_superuser | is_staff 
----+----------+--------------+----------
  2 | jsmith   | f            | t
  1 | root     | t            | t
(2 rows)

The old auth_user table is gone and cleanly replaced with users_user.

netbox=# select * from auth_group;
 id | name 
----+------
(0 rows)

netbox=# select * from users_group;
 id | name  | description 
----+-------+-------------
  1 | admin | 
(1 row)

⚠️⚠️ The old auth_group table is empty, but its contents seem to now be in users_group ⚠️⚠️

netbox=# select * from django_content_type where app_label in ('auth', 'users');
 id  | app_label |      model       
-----+-----------+------------------
   3 | auth      | permission
   4 | auth      | group
 115 | users     | userconfig
 116 | users     | token
 117 | users     | objectpermission
 140 | users     | group
   5 | users     | user
(7 rows)

The auth|user content type has been cleanly renamed to users|user.

⚠️⚠️ There are two group types auth|group and users|group ⚠️⚠️

netbox=# select * from auth_permission where content_type_id in (4, 5, 140);
 id  |       name       | content_type_id |   codename   
-----+------------------+-----------------+--------------
   9 | Can add group    |               4 | add_group
  10 | Can change group |               4 | change_group
  11 | Can delete group |               4 | delete_group
  12 | Can view group   |               4 | view_group
  13 | Can add user     |               5 | add_user
  14 | Can change user  |               5 | change_user
  15 | Can delete user  |               5 | delete_user
  16 | Can view user    |               5 | view_user
 558 | Can add group    |             140 | add_group
 559 | Can change group |             140 | change_group
 560 | Can delete group |             140 | delete_group
 561 | Can view group   |             140 | view_group
(12 rows)

⚠️⚠️ These two groups appear here too. ⚠️⚠️

netbox=# select * from users_objectpermission;
 id | name  | description | enabled |         actions          | constraints 
----+-------+-------------+---------+--------------------------+-------------
  1 | admin |             | t       | {view,add,change,delete} | 
(1 row)

Unchanged from 3.7.8

netbox=# select * from users_objectpermission_groups;
ERROR:  relation "users_objectpermission_groups" does not exist
LINE 1: select * from users_objectpermission_groups;
                      ^
netbox=# select * from users_group_object_permissions;
 id | objectpermission_id | group_id 
----+---------------------+----------
  1 |                   1 |        1
(1 row)

The old users_objectpermission_groups is gone and cleanly replaced with users_group_object_permissions

netbox=# select * from users_objectpermission_object_types;
 id | objectpermission_id | objecttype_id 
----+---------------------+---------------
  1 |                   1 |             4
  2 |                   1 |             5
(2 rows)

Unchanged from 3.7.8

[arthanson]

@ibuclaw Going through the other issues here - but FYI to clarify regarding the difference between auth.users and auth.groups remaining in the database. Django has support for custom user models, so when we replace users model the old one is replaced in the database (i.e. the old one is gone). For groups, Django doesn't have any built in support for replacing it so the old model will still be in the database (but un-used) along with the new one.