SystemZ: 128-bit element-type vector arguments unsupported

[aarnez]

The SystemZ backend currently doesn't support passing vectors with 128-bit element types such as long double or __int128. Verified with this snippet:

typedef long double sometype;

sometype get_elem(sometype __attribute__ ((vector_size (16))) v)
{
  return v[0];
}

When trying to compile, Clang crashes with this message:

fatal error: error in backend: Unsupported vector argument or return type
PLEASE submit a bug report…

The same happens when defining sometype to __int128 or when trying to return such a vector type instead of passing it as an argument.

[uweigand]

This is currently unsupported, but really should be fixed in LLVM. Thanks!

CC @JonPsson

[JonPsson]

@uweigand I see this was guarded against explicitly back in 2015 (5211f9f) with checks like "VerifyVectorType()". Do you have any thought on what are the problems relating to this? i128 is not a legal type in the backend - does that mean v1i128 could also not be legal, and is that the problem here?

[uweigand]

You're right that neither i128 nor v1i128 are legal types in the backend, and that contributes to the problem. However, although those are not legal types, it is still possible to correctly implement the ABI for those types, and we have done so for i128, but not for v1i128 - the latter is what we need to fix now.

Since i128 is not a legal type, it gets automatically split by common SelectionDAG code into two i64 arguments. This is wrong according to our ABI, and needs to be fixed by the back end. We do so using this clause here:

  // Same for i128 values.  These are already split into two i64 here,
  // so we have to use a custom handler.
  CCIfType<[i64], CCCustom<"CC_SystemZ_I128Indirect">>,

where the CC_SystemZ_I128Indirect handler inspects the original type behind the i64 part, and if it was i128, it does the right thing.

For v1i128, I believe (but you should verify) that common code also splits it into two i64 parts. We likewise would need to undo this, e.g. by having CC_SystemZ_I128Indirect handle this case as well, and do the right thing for this type. (i128 is passed by implicit reference in our ABI, but v1i128 is supposed to be passed in a VR.)