So I was having TemplateDoesNotExist errors when trying to set up everything.
It took me some time to realize that for this package to work, you need to have APP_DIRS: True for debug_toolbar to include its templates in the projects.
Perhaps it's a good idea to include this in the documentation?