Should batch consistency be tested with a batch of non-identical items?

[jcaw]

The batch_consistency_test.py files check that behaviour is consistent on batches vs. items, but the current infrastructure does this with batches of identical items. This approach may miss bugs related to data leakage between items within a batch, for example #994. I suggest items should be different to ensure those kinds of issues are flagged.

I have an alternate implementation here which uses batches with different items (where possible). Would you like me to open a PR?

(I've just implemented it for the functional tests for now, but I can also expand it to the root batch_consistency_check.py too.)

[mthrok]

Hi @jcaw

Thanks for the suggestion. I although thought a similar thing when reviewing #994. I briefly looked at your code and basically, your approach is to pass a batch to the helper method and run it separately and individually, correct? This looks like a good improvement. Please open a PR.

[jcaw]

Awesome. Yes, that's correct. I'll open a PR.