This is bound to cause problems at some point colliding with user code.
We have some prior art for vendoring Python packages in Keepsake that works pretty well. I'm happy to have a go at booting up that code / transferring that knowledge if we want to do this.
Migrated from a todo in #378.
