PREFER_IPV4 Setting is ignored in the sorting of tables by IP Address

[stuntguy3000]

Environment

• Python version: 3.6.8
• NetBox version: 2.10.4

Steps to Reproduce

1. Sort virtual machines or devices by IP Address
   https://[address]/virtualization/virtual-machines/?sort=primary_ip

Expected Behavior

IP Addresses will be displayed based on PREFER_IPV4 config setting.

Observed Behavior

IPv4 addresses are always shown.

Initially reported: #5820 (comment)