64bit offsets - LWT compatibility

[benjeffery]

We're moving to 64bit offset columns in #1527. This will create incompatibility in the LightweightTableCollection dict representation which is used to transfer a tree-sequence between python modules that have compiled different versions of the tskit C API.

There are two situations, a 64bit tskit reading a dict from a 32bit tskit, and vice-versa.

The first of these is relatively straight forward - we already use PyArray_FromAny which will silently apply any cast which is safe, so will read the 32bit offset array just fine.

The second case will fail as the cast is unsafe. My suggestion for dealing with this is that the 64bit code writes a 32bit array to any X_offset column if safe to do so and all the offsets are below the maximum value. If they aren't it writes 64bit and the 32bit code will error out. Unfortunately the error message will not be friendly, but there is no way to retrospectively fix that.

We should add some tests to CI with old msprime versions to test this.

[jeromekelleher]

This sounds good to me. So round-tripping through 32bit -> 64bit -> 32bit will work fine.

For the record, we also discussed keeping the current encoding where x_offset columns are always 32 bit, and putting in an extra x_offset_64 column. This proposal here is much better though.

So, when will this break? I guess if we have a forward simulation producing a table with large metadata, working with an old version of msprime. This will fail with an obscure error. @molpopgen, @petrelharp, any thoughts?

[benjeffery]

This sounds good to me. So round-tripping through 32bit -> 64bit -> 32bit will work fine.

Yep

For the record, we also discussed keeping the current encoding where x_offset columns are always 32 bit, and putting in an extra x_offset_64 column. This proposal here is much better though.

Yes, I gave it a bit more thought.

So, when will this break? I guess if we have a forward simulation producing a table with large metadata, working with an old version of msprime. This will fail with an obscure error. @molpopgen, @petrelharp, any thoughts?

I think breaking hard is preferable to truncating the data, which is the only other option.

[molpopgen]

The second case will fail as the cast is unsafe. My suggestion for dealing with this is that the 64bit code writes a 32bit array to any X_offset column if safe to do so and all the offsets are below the maximum value. If they aren't it writes 64bit and the 32bit code will error out.

To be honest, I'm having trouble following this explanation. For one, the safety implications--will it always fail (b/c of something about the Python function being used) or will it only sometimes fail when it encounters values out of range?

For the "if not case", will the pre-existing 32 bit values be "lifted over" to the 64 bit offset column, clearing out the first?

[benjeffery]

It will always fail as numpy refuses to cast down, it doesn't check contents, just dtype.

There are no pre-existing values, the LWT makes a new dict.

[molpopgen]

Thanks. I'm on board with this.

[hyanwong]

Unfortunately the error message will not be friendly, but there is no way to retrospectively fix that.

If the error is caused by casting in numpy, and is therefore on the python side, why can't you perform the cast within a try/except block, catch the error, and (if it is the relevant one) re-report it, changing the error to a more useful message?

[jeromekelleher]

The issue is that the error happens in code we've already shipped (e.g., msprime 1.0.0), so we can't change it.