[LV] Explicitly disable in-loop reductions for AnyOf and FindIV. nfc #163541

Mel-Chen · 2025-10-15T10:21:22Z

Currently, in-loop reductions for AnyOf and FindIV are not supported.
They were implicitly blocked. This happened because RecurrenceDescriptor::getReductionOpChain could not detect their recurrence chain. The reason is that RecurrenceDescriptor::getOpcode was set to Instruction::Or, but the recurrence chains of AnyOf and FindIV do not actually contain an Instruction::Or.

This patch explicitly disables in-loop reductions for AnyOf and FindIV instead of relying on getReductionOpChain to implicitly prevent them.

llvmbot · 2025-10-15T10:22:00Z

@llvm/pr-subscribers-vectorizers

@llvm/pr-subscribers-llvm-transforms

Author: Mel Chen (Mel-Chen)

Changes

Full diff: https://github.com/llvm/llvm-project/pull/163541.diff

3 Files Affected:

(modified) llvm/lib/Analysis/IVDescriptors.cpp (+7-5)
(modified) llvm/lib/Transforms/Vectorize/LoopVectorize.cpp (+6-1)
(modified) llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp (+3-4)

diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 9f8ac6e8e2e0b..641850b46bbd8 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1220,11 +1220,6 @@ unsigned RecurrenceDescriptor::getOpcode(RecurKind Kind) {
     return Instruction::Add;
   case RecurKind::Mul:
     return Instruction::Mul;
-  case RecurKind::AnyOf:
-  case RecurKind::FindFirstIVSMin:
-  case RecurKind::FindFirstIVUMin:
-  case RecurKind::FindLastIVSMax:
-  case RecurKind::FindLastIVUMax:
   case RecurKind::Or:
     return Instruction::Or;
   case RecurKind::And:
@@ -1248,6 +1243,13 @@ unsigned RecurrenceDescriptor::getOpcode(RecurKind Kind) {
   case RecurKind::FMaximumNum:
   case RecurKind::FMinimumNum:
     return Instruction::FCmp;
+  case RecurKind::AnyOf:
+  case RecurKind::FindFirstIVSMin:
+  case RecurKind::FindFirstIVUMin:
+  case RecurKind::FindLastIVSMax:
+  case RecurKind::FindLastIVUMax:
+    // TODO: Set AnyOf and FindIV to Instruction::Select once in-loop reductions
+    // are supported.
   default:
     llvm_unreachable("Unknown recurrence operation");
   }
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index b96d29e635465..77916a0ea744d 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -6515,9 +6515,14 @@ void LoopVectorizationCostModel::collectInLoopReductions() {
     if (RdxDesc.getRecurrenceType() != Phi->getType())
       continue;
 
+    // In-loop AnyOf and FindIV reductions are not yet supported.
+    RecurKind Kind = RdxDesc.getRecurrenceKind();
+    if (RecurrenceDescriptor::isAnyOfRecurrenceKind(Kind) ||
+        RecurrenceDescriptor::isFindIVRecurrenceKind(Kind))
+      continue;
+
     // If the target would prefer this reduction to happen "in-loop", then we
     // want to record it as such.
-    RecurKind Kind = RdxDesc.getRecurrenceKind();
     if (!PreferInLoopReductions && !useOrderedReductions(RdxDesc) &&
         !TTI.preferInLoopReduction(Kind, Phi->getType()))
       continue;
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index 2368d18b0373c..95d0b899b9085 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -784,10 +784,9 @@ Value *VPInstruction::generate(VPTransformState &State) {
     auto *OrigPhi = cast<PHINode>(PhiR->getUnderlyingValue());
     Value *ReducedPartRdx = State.get(getOperand(2));
     for (unsigned Idx = 3; Idx < getNumOperands(); ++Idx)
-      ReducedPartRdx = Builder.CreateBinOp(
-          (Instruction::BinaryOps)RecurrenceDescriptor::getOpcode(
-              RecurKind::AnyOf),
-          State.get(getOperand(Idx)), ReducedPartRdx, "bin.rdx");
+      ReducedPartRdx =
+          Builder.CreateBinOp(Instruction::Or, State.get(getOperand(Idx)),
+                              ReducedPartRdx, "bin.rdx");
     return createAnyOfReduction(Builder, ReducedPartRdx,
                                 State.get(getOperand(1), VPLane(0)), OrigPhi);
   }

llvmbot · 2025-10-15T10:22:01Z

@llvm/pr-subscribers-llvm-analysis

Author: Mel Chen (Mel-Chen)

Changes

Full diff: https://github.com/llvm/llvm-project/pull/163541.diff

3 Files Affected:

(modified) llvm/lib/Analysis/IVDescriptors.cpp (+7-5)
(modified) llvm/lib/Transforms/Vectorize/LoopVectorize.cpp (+6-1)
(modified) llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp (+3-4)

diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 9f8ac6e8e2e0b..641850b46bbd8 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1220,11 +1220,6 @@ unsigned RecurrenceDescriptor::getOpcode(RecurKind Kind) {
     return Instruction::Add;
   case RecurKind::Mul:
     return Instruction::Mul;
-  case RecurKind::AnyOf:
-  case RecurKind::FindFirstIVSMin:
-  case RecurKind::FindFirstIVUMin:
-  case RecurKind::FindLastIVSMax:
-  case RecurKind::FindLastIVUMax:
   case RecurKind::Or:
     return Instruction::Or;
   case RecurKind::And:
@@ -1248,6 +1243,13 @@ unsigned RecurrenceDescriptor::getOpcode(RecurKind Kind) {
   case RecurKind::FMaximumNum:
   case RecurKind::FMinimumNum:
     return Instruction::FCmp;
+  case RecurKind::AnyOf:
+  case RecurKind::FindFirstIVSMin:
+  case RecurKind::FindFirstIVUMin:
+  case RecurKind::FindLastIVSMax:
+  case RecurKind::FindLastIVUMax:
+    // TODO: Set AnyOf and FindIV to Instruction::Select once in-loop reductions
+    // are supported.
   default:
     llvm_unreachable("Unknown recurrence operation");
   }
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index b96d29e635465..77916a0ea744d 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -6515,9 +6515,14 @@ void LoopVectorizationCostModel::collectInLoopReductions() {
     if (RdxDesc.getRecurrenceType() != Phi->getType())
       continue;
 
+    // In-loop AnyOf and FindIV reductions are not yet supported.
+    RecurKind Kind = RdxDesc.getRecurrenceKind();
+    if (RecurrenceDescriptor::isAnyOfRecurrenceKind(Kind) ||
+        RecurrenceDescriptor::isFindIVRecurrenceKind(Kind))
+      continue;
+
     // If the target would prefer this reduction to happen "in-loop", then we
     // want to record it as such.
-    RecurKind Kind = RdxDesc.getRecurrenceKind();
     if (!PreferInLoopReductions && !useOrderedReductions(RdxDesc) &&
         !TTI.preferInLoopReduction(Kind, Phi->getType()))
       continue;
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index 2368d18b0373c..95d0b899b9085 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -784,10 +784,9 @@ Value *VPInstruction::generate(VPTransformState &State) {
     auto *OrigPhi = cast<PHINode>(PhiR->getUnderlyingValue());
     Value *ReducedPartRdx = State.get(getOperand(2));
     for (unsigned Idx = 3; Idx < getNumOperands(); ++Idx)
-      ReducedPartRdx = Builder.CreateBinOp(
-          (Instruction::BinaryOps)RecurrenceDescriptor::getOpcode(
-              RecurKind::AnyOf),
-          State.get(getOperand(Idx)), ReducedPartRdx, "bin.rdx");
+      ReducedPartRdx =
+          Builder.CreateBinOp(Instruction::Or, State.get(getOperand(Idx)),
+                              ReducedPartRdx, "bin.rdx");
     return createAnyOfReduction(Builder, ReducedPartRdx,
                                 State.get(getOperand(1), VPLane(0)), OrigPhi);
   }

fhahn

Thanks!

Could you also update the description of the PR to clarify why this is NFC?

fhahn · 2025-10-15T10:50:23Z

llvm/lib/Analysis/IVDescriptors.cpp

  case RecurKind::FMaximumNum:
  case RecurKind::FMinimumNum:
    return Instruction::FCmp;
+  case RecurKind::AnyOf:


Could we still leave AnyOf to return Or to avoid the changes in llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp?

I think we should maintain consistency in how we handle AnyOf and FindIV. In particular, AnyOf, whether it currently only supports integer types or will support floating-point types in the future, should consistently use Instruction::Or in ComputeAnyOfResult to fold each part.

Ok sounds good to me.

Mel-Chen · 2025-11-12T07:56:28Z

Ping, or we can go #162252

fhahn

LGTM, thanks

fhahn · 2025-11-12T09:36:46Z

llvm/lib/Analysis/IVDescriptors.cpp

  case RecurKind::FMaximumNum:
  case RecurKind::FMinimumNum:
    return Instruction::FCmp;
+  case RecurKind::AnyOf:


Ok sounds good to me.

ayalz

Couple of post-commit review comments.

ayalz · 2025-11-15T11:04:36Z

llvm/lib/Analysis/IVDescriptors.cpp

+    // TODO: Set AnyOf and FindIV to Instruction::Select once in-loop reductions
+    // are supported.


Suggested change

// TODO: Set AnyOf and FindIV to Instruction::Select once in-loop reductions

// are supported.

// TODO: Set AnyOf and FindIV to Instruction::Select once in-loop reductions

// are supported.

llvm_unreachable("AnyOf and FindIV do not support in-loop");

rather than claiming they are Unknown recurrence operations?

Should the enclosing getOpcode() be renamed getInLoopOpcode()?

ayalz · 2025-11-15T11:14:55Z

llvm/lib/Analysis/IVDescriptors.cpp

+  case RecurKind::FindFirstIVUMin:
+  case RecurKind::FindLastIVSMax:
+  case RecurKind::FindLastIVUMax:
+    // TODO: Set AnyOf and FindIV to Instruction::Select once in-loop reductions


The reason is that RecurrenceDescriptor::getOpcode was set to Instruction::Or, but the recurrence chains of AnyOf and FindIV do not actually contain an Instruction::Or.

Boolean recurrence chain of AnyOf could indeed contain a Select as in nextRed = currentCond ? true : prevRed, but could also contain an Or as in nextRed = currentCond | prevRed?

Integer recurrence chain of FindIV indeed contains a Select: nextRed = currentCond ? currentIV : prevRed, though it would be good to clarify when the opcode is Select rather than an ICmp/FCmp feeding it - as in min/max reductions whose chains may also contain a Select.

Suffice for now to leave behind a TODO to return a valid opcode once these kinds of recurrences support in-loop reduction.

Mel-Chen requested review from artagnon, ayalz, fhahn and huntergr-arm October 15, 2025 10:21

llvmbot added vectorizers llvm:analysis Includes value tracking, cost tables and constant folding llvm:transforms labels Oct 15, 2025

Mel-Chen mentioned this pull request Oct 15, 2025

[LV] Change opcode of FindLast/FindFirst recurrence to Instruction::Select #162252

Closed

fhahn reviewed Oct 15, 2025

View reviewed changes

fhahn approved these changes Nov 12, 2025

View reviewed changes

[LV] Explicitly disable in-loop reductions for AnyOf and FindIV

15baf5f

Mel-Chen force-pushed the nfc-unreachable-opcode branch from e64030f to 15baf5f Compare November 14, 2025 08:45

Mel-Chen enabled auto-merge (squash) November 14, 2025 08:45

Mel-Chen merged commit 3277f6c into llvm:main Nov 14, 2025
7 of 9 checks passed

ayalz reviewed Nov 15, 2025

View reviewed changes

		// TODO: Set AnyOf and FindIV to Instruction::Select once in-loop reductions
		// are supported.

[LV] Explicitly disable in-loop reductions for AnyOf and FindIV. nfc #163541

[LV] Explicitly disable in-loop reductions for AnyOf and FindIV. nfc #163541

Conversation

Mel-Chen commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented Oct 15, 2025

Uh oh!

fhahn left a comment

Choose a reason for hiding this comment

Uh oh!

fhahn Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Mel-Chen Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

fhahn Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

Mel-Chen commented Nov 12, 2025

Uh oh!

fhahn left a comment

Choose a reason for hiding this comment

Uh oh!

fhahn Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ayalz left a comment

Choose a reason for hiding this comment

Uh oh!

ayalz Nov 15, 2025

Choose a reason for hiding this comment

Uh oh!

ayalz Nov 15, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Mel-Chen commented Oct 15, 2025 •

edited

Loading

llvmbot commented Oct 15, 2025 •

edited

Loading