changing the way core files are stored in SmartOS VMs so that we can actually use them #492
Description
This is related to nodejs/node#7649, where a test running on SmartOS made a node process abort, and thus made the system generate a core file.
That core file could have helped us root-causing the problem, but unfortunately it wasn't available anymore because by default core files are stored in the global zone of the server on which the VM runs, and they're deleted after one week.
Another problem with the default setup for core file storage with SmartOS test VMs is that only Triton cloud's operators can access core files stored in the server's global zone.
What we could do is set the configuration of every SmartOS test VM so that:
- core files are stored somewhere that build WG members can access
- core files are cleaned up after some time (e.g 1 week) by a cron job to avoid filling up space
This way, when a test failure happens due to a node process aborting on a SmartOS test VM, the person who ran the CI tests job can ask a member of the build WG to get the core file and e.g upload it to manta so that it can be inspected with mdb_v8.
Regardless of whether the file needs to be inspected, it would be deleted after a week, and not fill up space on test VMs.
Does that sound like a useful thing to do? If so I can help set it up, just let me know.