[SPARK-38313] Remove deprecated logic in SortMergeJoinExec

[leesf]

What changes were proposed in this pull request?

Remove deprecated logic in SortMergeJoinExec in full outer sort merge, while the PR handles compare result separately, so the original code to handle < logic would be removed.

Why are the changes needed?

Make code cleaner

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing tests.

[leesf]

@c21 @cloud-fan would you please help to review the PR when you have time? Thanks in advance.

[c21]

I don't think it's safe to remove leftMatched/rightMatched. These two bit sets are used to record whether the rows in leftMatches/rightMatches has matched row or not. For unmatched rows, FULL OUTER join still needs to output one row with null values set on the other side. We still needs the bit sets here because rows in buffers (leftMatches/rightMatches) are having same join keys values, but may still not match because the join condition needs to be passed.

[leesf]

join still needs to output one row

@c21 Thanks for the comment. However in which case will it exists unmatched rows? rows in leftMatches/rightMatches are the same, so it should be all matched. And also would you please explain more about the but may still not match because the join condition needs to be passed.

[cloud-fan]

There are test failures, looks like this change breaks some queries.

[c21]

However in which case will it exists unmatched rows?

@leesf - Beside the equi join condition, we also have non-equi condition. e.g.

SELECT *
FROM t1
JOIN t2
ON 
  t1.k = t2.k
  AND t1.v != t2.v     <- this is the non-equi condition

So rows in left, right buffers have same values for join keys (i.e. k column here), but not necessarily pass the non-equi join condition (i.e. v column here). SortMergeJoinExec.condition is the non-equi condition (i.e. t1.v != t2.v here).

[AmplabJenkins]

Can one of the admins verify this patch?

[leesf]

However in which case will it exists unmatched rows?

@leesf - Beside the equi join condition, we also have non-equi condition. e.g.

SELECT *
FROM t1
JOIN t2
ON 
  t1.k = t2.k
  AND t1.v != t2.v     <- this is the non-equi condition

So rows in left, right buffers have same values for join keys (i.e. k column here), but not necessarily pass the non-equi join condition (i.e. v column here). SortMergeJoinExec.condition is the non-equi condition (i.e. t1.v != t2.v here).

Thanks for you explanation and example, you are right and i am going to close the PR.