Follow-up of #555. We should also add the `setup-proxy-cache` to the test workflows, now that we could be downloading lots of wheels during testing.
