[delinearize] remove all uses of GEP instructions #156215
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
llvmbot
added
llvm:analysis
|
@llvm/pr-subscribers-llvm-analysis Author: Sebastian Pop (sebpop) ChangesThis patch series removes the use of GEP instructions from delinearization.
Testcases adjusted to maintain the patterns they test for. The third patch can be rewritten (if alloca's and globals are also going to be scheduled for removal from LLVM IR) to send the data from the front-ends attached to The current patch does not change the LLVM IR, it assumes the array info can be discovered through inter-procedural analysis, and if IPA is not available, it relies on the other symbolic delinearization techniques. Patch is 225.80 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/156215.diff 93 Files Affected:
diff --git a/llvm/include/llvm/Analysis/Delinearization.h b/llvm/include/llvm/Analysis/Delinearization.h
index 434cfb61699d6..2eda019abaf46 100644
--- a/llvm/include/llvm/Analysis/Delinearization.h
+++ b/llvm/include/llvm/Analysis/Delinearization.h
@@ -112,6 +112,13 @@ void delinearize(ScalarEvolution &SE, const SCEV *Expr,
SmallVectorImpl<const SCEV *> &Subscripts,
SmallVectorImpl<const SCEV *> &Sizes, const SCEV *ElementSize);
+/// Same as delinearize. TODO: only use delinearize() instead of other internal
+/// functions.
+bool delinearizeUsingArrayInfo(ScalarEvolution &SE, const SCEV *AccessFn,
+ SmallVectorImpl<const SCEV *> &Subscripts,
+ SmallVectorImpl<const SCEV *> &Sizes,
+ const SCEV *ElementSize);
+
/// Compute the dimensions of fixed size array from \Expr and save the results
/// in \p Sizes.
bool findFixedSizeArrayDimensions(ScalarEvolution &SE, const SCEV *Expr,
@@ -132,28 +139,21 @@ bool findFixedSizeArrayDimensions(ScalarEvolution &SE, const SCEV *Expr,
/// Different from the normal delinearization, this function assumes that NO
/// terms exist in the \p Expr. In other words, it assumes that the all step
/// values are constant.
-///
-/// This function is intended to replace getIndexExpressionsFromGEP and
-/// tryDelinearizeFixedSizeImpl. They rely on the GEP source element type so
-/// that they will be removed in the future.
bool delinearizeFixedSizeArray(ScalarEvolution &SE, const SCEV *Expr,
SmallVectorImpl<const SCEV *> &Subscripts,
SmallVectorImpl<const SCEV *> &Sizes,
const SCEV *ElementSize);
-/// Gathers the individual index expressions from a GEP instruction.
-///
-/// This function optimistically assumes the GEP references into a fixed size
-/// array. If this is actually true, this function returns a list of array
-/// subscript expressions in \p Subscripts and a list of integers describing
-/// the size of the individual array dimensions in \p Sizes. Both lists have
-/// either equal length or the size list is one element shorter in case there
-/// is no known size available for the outermost array dimension. Returns true
-/// if successful and false otherwise.
-bool getIndexExpressionsFromGEP(ScalarEvolution &SE,
- const GetElementPtrInst *GEP,
- SmallVectorImpl<const SCEV *> &Subscripts,
- SmallVectorImpl<int> &Sizes);
+/// Compute access functions for each subscript in a delinearized array access.
+void computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
+ SmallVectorImpl<const SCEV *> &Subscripts,
+ SmallVectorImpl<const SCEV *> &Sizes,
+ const SCEV *ElementSize);
+
+/// Backward compatibility wrapper for computeAccessFunctions.
+void computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
+ SmallVectorImpl<const SCEV *> &Subscripts,
+ SmallVectorImpl<const SCEV *> &Sizes);
/// Implementation of fixed size array delinearization. Try to delinearize
/// access function for a fixed size multi-dimensional array, by deriving
diff --git a/llvm/lib/Analysis/Delinearization.cpp b/llvm/lib/Analysis/Delinearization.cpp
index 762d9191aab1e..873060d5f7f2a 100644
--- a/llvm/lib/Analysis/Delinearization.cpp
+++ b/llvm/lib/Analysis/Delinearization.cpp
@@ -14,15 +14,18 @@
//===----------------------------------------------------------------------===//
#include "llvm/Analysis/Delinearization.h"
+#include "llvm/ADT/DenseMap.h"
#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/ScalarEvolution.h"
#include "llvm/Analysis/ScalarEvolutionDivision.h"
#include "llvm/Analysis/ScalarEvolutionExpressions.h"
#include "llvm/IR/Constants.h"
+#include "llvm/IR/DataLayout.h"
#include "llvm/IR/DerivedTypes.h"
#include "llvm/IR/Function.h"
#include "llvm/IR/InstIterator.h"
#include "llvm/IR/Instructions.h"
+#include "llvm/IR/Module.h"
#include "llvm/IR/PassManager.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Debug.h"
@@ -182,7 +185,7 @@ void llvm::collectParametricTerms(ScalarEvolution &SE, const SCEV *Expr,
LLVM_DEBUG({
dbgs() << "Strides:\n";
for (const SCEV *S : Strides)
- dbgs() << *S << "\n";
+ dbgs() << " " << *S << "\n";
});
for (const SCEV *S : Strides) {
@@ -193,7 +196,7 @@ void llvm::collectParametricTerms(ScalarEvolution &SE, const SCEV *Expr,
LLVM_DEBUG({
dbgs() << "Terms:\n";
for (const SCEV *T : Terms)
- dbgs() << *T << "\n";
+ dbgs() << " " << *T << "\n";
});
SCEVCollectAddRecMultiplies MulCollector(Terms, SE);
@@ -294,7 +297,7 @@ void llvm::findArrayDimensions(ScalarEvolution &SE,
LLVM_DEBUG({
dbgs() << "Terms:\n";
for (const SCEV *T : Terms)
- dbgs() << *T << "\n";
+ dbgs() << " " << *T << "\n";
});
// Remove duplicates.
@@ -325,7 +328,7 @@ void llvm::findArrayDimensions(ScalarEvolution &SE,
LLVM_DEBUG({
dbgs() << "Terms after sorting:\n";
for (const SCEV *T : NewTerms)
- dbgs() << *T << "\n";
+ dbgs() << " " << *T << "\n";
});
if (NewTerms.empty() || !findArrayDimensionsRec(SE, NewTerms, Sizes)) {
@@ -339,13 +342,14 @@ void llvm::findArrayDimensions(ScalarEvolution &SE,
LLVM_DEBUG({
dbgs() << "Sizes:\n";
for (const SCEV *S : Sizes)
- dbgs() << *S << "\n";
+ dbgs() << " " << *S << "\n";
});
}
void llvm::computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
SmallVectorImpl<const SCEV *> &Subscripts,
- SmallVectorImpl<const SCEV *> &Sizes) {
+ SmallVectorImpl<const SCEV *> &Sizes,
+ const SCEV *ElementSize) {
// Early exit in case this SCEV is not an affine multivariate function.
if (Sizes.empty())
return;
@@ -354,20 +358,32 @@ void llvm::computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
if (!AR->isAffine())
return;
+ if (ElementSize->isZero())
+ return;
+
+ // Clear output vector.
+ Subscripts.clear();
+
+ LLVM_DEBUG(dbgs() << "\ncomputeAccessFunctions\n"
+ << "Linearized Memory Access Function: " << *Expr << "\n");
+
const SCEV *Res = Expr;
int Last = Sizes.size() - 1;
+
for (int i = Last; i >= 0; i--) {
- const SCEV *Q, *R;
- SCEVDivision::divide(SE, Res, Sizes[i], &Q, &R);
+ const SCEV *Size = Sizes[i];
+ if (Size->isZero())
+ continue;
+ const SCEV *Q, *R;
+ SCEVDivision::divide(SE, Res, Size, &Q, &R);
LLVM_DEBUG({
- dbgs() << "Res: " << *Res << "\n";
- dbgs() << "Sizes[i]: " << *Sizes[i] << "\n";
- dbgs() << "Res divided by Sizes[i]:\n";
- dbgs() << "Quotient: " << *Q << "\n";
- dbgs() << "Remainder: " << *R << "\n";
+ dbgs() << "Computing 'MemAccFn / Sizes[" << i << "]':\n";
+ dbgs() << " MemAccFn: " << *Res << "\n";
+ dbgs() << " Sizes[" << i << "]: " << *Size << "\n";
+ dbgs() << " Quotient (Leftover): " << *Q << "\n";
+ dbgs() << " Remainder (Subscript Access Function): " << *R << "\n";
});
-
Res = Q;
// Do not record the last subscript corresponding to the size of elements in
@@ -385,22 +401,162 @@ void llvm::computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
}
// Record the access function for the current subscript.
+ LLVM_DEBUG(dbgs() << "Subscripts push_back Remainder: " << *R << "\n");
Subscripts.push_back(R);
}
- // Also push in last position the remainder of the last division: it will be
- // the access function of the innermost dimension.
- Subscripts.push_back(Res);
+ // Also push in last position the quotient "Res = Q" of the last division: it
+ // will be the access function of the outermost array dimension.
+ if (!Res->isZero()) {
+ // This is only needed when the outermost array size is not known. Res = 0
+ // when the outermost array dimension is known, as for example when reading
+ // array sizes from a local or global declaration.
+ Subscripts.push_back(Res);
+ LLVM_DEBUG(dbgs() << "Subscripts push_back Res: " << *Res << "\n");
+ }
std::reverse(Subscripts.begin(), Subscripts.end());
LLVM_DEBUG({
dbgs() << "Subscripts:\n";
for (const SCEV *S : Subscripts)
- dbgs() << *S << "\n";
+ dbgs() << " " << *S << "\n";
+ dbgs() << "\n";
});
}
+/// Backward compatibility wrapper for the old 4-parameter version.
+void llvm::computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
+ SmallVectorImpl<const SCEV *> &Subscripts,
+ SmallVectorImpl<const SCEV *> &Sizes) {
+ // Use the element size from the last element in Sizes array (legacy behavior)
+ if (Sizes.empty()) {
+ Subscripts.clear();
+ return;
+ }
+ const SCEV *ElementSize = Sizes.back();
+ computeAccessFunctions(SE, Expr, Subscripts, Sizes, ElementSize);
+}
+
+/// Extract array dimensions from alloca or global variable declarations.
+/// Returns true if array dimensions were successfully extracted.
+static bool
+extractArrayInfoFromAllocaOrGlobal(ScalarEvolution &SE, Value *BasePtr,
+ SmallVectorImpl<const SCEV *> &Sizes,
+ const SCEV *ElementSize) {
+ // Clear output vector.
+ Sizes.clear();
+
+ LLVM_DEBUG(dbgs() << "extractArrayInfoFromAllocaOrGlobal called with BasePtr: " << *BasePtr << "\n");
+
+ // Distinguish between simple array accesses and complex pointer arithmetic.
+ // Only apply array_info extraction to direct array accesses to avoid incorrect
+ // delinearization of complex pointer arithmetic patterns.
+ if (auto *GEP = dyn_cast<GetElementPtrInst>(BasePtr)) {
+ // Check if this is a simple array access pattern: GEP [N x T]* @array, 0, idx
+ // This represents direct indexing like array[i], which should use array dimensions.
+ if (GEP->getNumIndices() == 2) {
+ auto *FirstIdx = dyn_cast<ConstantInt>(GEP->getOperand(1));
+ if (FirstIdx && FirstIdx->isZero()) {
+ // Simple array access: extract dimensions from the underlying array type
+ Value *Source = GEP->getPointerOperand()->stripPointerCasts();
+ return extractArrayInfoFromAllocaOrGlobal(SE, Source, Sizes, ElementSize);
+ }
+ }
+ // Complex GEPs like (&array[offset])[index] represent pointer arithmetic, not
+ // simple array indexing. These should be handled by parametric delinearization
+ // to preserve the linearized byte-offset semantics rather than treating them
+ // as multidimensional array accesses.
+ return false;
+ }
+
+ // Check if BasePtr is from an alloca instruction.
+ Type *ElementType = nullptr;
+ if (auto *AI = dyn_cast<AllocaInst>(BasePtr)) {
+ ElementType = AI->getAllocatedType();
+ LLVM_DEBUG(dbgs() << "Found alloca with type: " << *ElementType << "\n");
+ } else if (auto *GV = dyn_cast<GlobalVariable>(BasePtr)) {
+ ElementType = GV->getValueType();
+ LLVM_DEBUG(dbgs() << "Found global variable with type: " << *ElementType
+ << "\n");
+ } else {
+ LLVM_DEBUG(dbgs() << "No alloca or global found for base pointer\n");
+ return false;
+ }
+
+ // Extract dimensions from nested array types.
+ Type *I64Ty = Type::getInt64Ty(SE.getContext());
+
+ while (auto *ArrayTy = dyn_cast<ArrayType>(ElementType)) {
+ uint64_t Size = ArrayTy->getNumElements();
+ const SCEV *SizeSCEV = SE.getConstant(I64Ty, Size);
+ Sizes.push_back(SizeSCEV);
+ ElementType = ArrayTy->getElementType();
+ LLVM_DEBUG(dbgs() << " Found array dimension: " << Size << "\n");
+ }
+
+ if (Sizes.empty()) {
+ LLVM_DEBUG(dbgs() << "No array dimensions found in type\n");
+ return false;
+ }
+
+ // Add element size as the last element for computeAccessFunctions algorithm.
+ Sizes.push_back(ElementSize);
+
+ LLVM_DEBUG({
+ dbgs() << "Extracted array info from alloca/global for base pointer "
+ << *BasePtr << "\n";
+ dbgs() << "Dimensions: ";
+ for (const SCEV *Size : Sizes)
+ dbgs() << *Size << " ";
+ dbgs() << "\n";
+ });
+
+ return true;
+}
+
+bool llvm::delinearizeUsingArrayInfo(ScalarEvolution &SE, const SCEV *AccessFn,
+ SmallVectorImpl<const SCEV *> &Subscripts,
+ SmallVectorImpl<const SCEV *> &Sizes,
+ const SCEV *ElementSize) {
+ // Clear output vectors.
+ Subscripts.clear();
+ Sizes.clear();
+
+ const SCEVUnknown *BasePointer =
+ dyn_cast<SCEVUnknown>(SE.getPointerBase(AccessFn));
+ if (!BasePointer) {
+ LLVM_DEBUG(dbgs() << "no BasePointer for AccessFn: " << *AccessFn << "\n");
+ return false;
+ }
+
+ Value *BasePtr = BasePointer->getValue();
+
+ // Extract array dimensions from alloca or global declarations.
+ if (!extractArrayInfoFromAllocaOrGlobal(SE, BasePtr, Sizes, ElementSize))
+ return false;
+
+ // Get the full SCEV expression and subtract the base pointer to get
+ // offset-only expression.
+ const SCEV *Expr = SE.getMinusSCEV(AccessFn, BasePointer);
+
+ computeAccessFunctions(SE, Expr, Subscripts, Sizes, ElementSize);
+ if (Sizes.empty() || Subscripts.empty())
+ return false;
+
+ // Validate dimension consistency: subscripts should match array dimensions
+ // (Sizes includes element size as last element, so array dimensions = Sizes.size() - 1)
+ unsigned ArrayDims = Sizes.size() - 1;
+ if (Subscripts.size() != ArrayDims) {
+ LLVM_DEBUG(dbgs() << "delinearizeUsingArrayInfo: Dimension mismatch - "
+ << Subscripts.size() << " subscripts for " << ArrayDims
+ << " array dimensions. Falling back to parametric method.\n");
+ return false;
+ }
+
+ return true;
+}
+
/// Splits the SCEV into two vectors of SCEVs representing the subscripts and
/// sizes of an array access. Returns the remainder of the delinearization that
/// is the offset start of the array. The SCEV->delinearize algorithm computes
@@ -454,36 +610,35 @@ void llvm::delinearize(ScalarEvolution &SE, const SCEV *Expr,
SmallVectorImpl<const SCEV *> &Subscripts,
SmallVectorImpl<const SCEV *> &Sizes,
const SCEV *ElementSize) {
- // First step: collect parametric terms.
+ // Clear output vectors.
+ Subscripts.clear();
+ Sizes.clear();
+
+ // Try array_info extraction.
+ if (delinearizeUsingArrayInfo(SE, Expr, Subscripts, Sizes, ElementSize))
+ return;
+ LLVM_DEBUG(dbgs() << "delinearize falling back to parametric method\n");
+
+ // Fall back to parametric delinearization.
+ const SCEVUnknown *BasePointer =
+ dyn_cast<SCEVUnknown>(SE.getPointerBase(Expr));
+ if (BasePointer)
+ Expr = SE.getMinusSCEV(Expr, BasePointer);
+
SmallVector<const SCEV *, 4> Terms;
collectParametricTerms(SE, Expr, Terms);
if (Terms.empty())
return;
- // Second step: find subscript sizes.
+ // Find subscript sizes.
findArrayDimensions(SE, Terms, Sizes, ElementSize);
if (Sizes.empty())
return;
- // Third step: compute the access functions for each subscript.
- computeAccessFunctions(SE, Expr, Subscripts, Sizes);
-
- if (Subscripts.empty())
- return;
-
- LLVM_DEBUG({
- dbgs() << "succeeded to delinearize " << *Expr << "\n";
- dbgs() << "ArrayDecl[UnknownSize]";
- for (const SCEV *S : Sizes)
- dbgs() << "[" << *S << "]";
-
- dbgs() << "\nArrayRef";
- for (const SCEV *S : Subscripts)
- dbgs() << "[" << *S << "]";
- dbgs() << "\n";
- });
+ // Compute the access functions for each subscript.
+ computeAccessFunctions(SE, Expr, Subscripts, Sizes, ElementSize);
}
static std::optional<APInt> tryIntoAPInt(const SCEV *S) {
@@ -492,6 +647,21 @@ static std::optional<APInt> tryIntoAPInt(const SCEV *S) {
return std::nullopt;
}
+/// Convert cached SCEV sizes to int sizes for compatibility.
+/// TODO: Remove this after we remove GEP delinearization.
+static void convertSCEVSizesToIntSizes(ArrayRef<const SCEV *> SCEVSizes,
+ SmallVectorImpl<int> &Sizes) {
+ for (const SCEV *S : SCEVSizes) {
+ if (auto *Const = dyn_cast<SCEVConstant>(S)) {
+ const APInt &APVal = Const->getAPInt();
+ if (APVal.isSignedIntN(32)) {
+ int intValue = APVal.getSExtValue();
+ Sizes.push_back(intValue);
+ }
+ }
+ }
+}
+
/// Collects the absolute values of constant steps for all induction variables.
/// Returns true if we can prove that all step recurrences are constants and \p
/// Expr is divisible by \p ElementSize. Each step recurrence is stored in \p
@@ -646,6 +816,9 @@ bool llvm::delinearizeFixedSizeArray(ScalarEvolution &SE, const SCEV *Expr,
SmallVectorImpl<const SCEV *> &Subscripts,
SmallVectorImpl<const SCEV *> &Sizes,
const SCEV *ElementSize) {
+ // Clear output vectors.
+ Subscripts.clear();
+ Sizes.clear();
// First step: find the fixed array size.
SmallVector<uint64_t, 4> ConstSizes;
@@ -659,52 +832,18 @@ bool llvm::delinearizeFixedSizeArray(ScalarEvolution &SE, const SCEV *Expr,
Sizes.push_back(SE.getConstant(Expr->getType(), Size));
// Second step: compute the access functions for each subscript.
- computeAccessFunctions(SE, Expr, Subscripts, Sizes);
+ computeAccessFunctions(SE, Expr, Subscripts, Sizes, ElementSize);
return !Subscripts.empty();
}
-bool llvm::getIndexExpressionsFromGEP(ScalarEvolution &SE,
- const GetElementPtrInst *GEP,
- SmallVectorImpl<const SCEV *> &Subscripts,
- SmallVectorImpl<int> &Sizes) {
- assert(Subscripts.empty() && Sizes.empty() &&
- "Expected output lists to be empty on entry to this function.");
- assert(GEP && "getIndexExpressionsFromGEP called with a null GEP");
- Type *Ty = nullptr;
- bool DroppedFirstDim = false;
- for (unsigned i = 1; i < GEP->getNumOperands(); i++) {
- const SCEV *Expr = SE.getSCEV(GEP->getOperand(i));
- if (i == 1) {
- Ty = GEP->getSourceElementType();
- if (auto *Const = dyn_cast<SCEVConstant>(Expr))
- if (Const->getValue()->isZero()) {
- DroppedFirstDim = true;
- continue;
- }
- Subscripts.push_back(Expr);
- continue;
- }
-
- auto *ArrayTy = dyn_cast<ArrayType>(Ty);
- if (!ArrayTy) {
- Subscripts.clear();
- Sizes.clear();
- return false;
- }
-
- Subscripts.push_back(Expr);
- if (!(DroppedFirstDim && i == 2))
- Sizes.push_back(ArrayTy->getNumElements());
-
- Ty = ArrayTy->getElementType();
- }
- return !Subscripts.empty();
-}
-
bool llvm::tryDelinearizeFixedSizeImpl(
ScalarEvolution *SE, Instruction *Inst, const SCEV *AccessFn,
SmallVectorImpl<const SCEV *> &Subscripts, SmallVectorImpl<int> &Sizes) {
+ // Clear output vectors.
+ Subscripts.clear();
+ Sizes.clear();
+
Value *SrcPtr = getLoadStorePointerOperand(Inst);
// Check the simple case where the array dimensions are fixed size.
@@ -712,31 +851,17 @@ bool llvm::tryDelinearizeFixedSizeImpl(
if (!SrcGEP)
return false;
- getIndexExpressionsFromGEP(*SE, SrcGEP, Subscripts, Sizes);
-
- // Check that the two size arrays are non-empty and equal in length and
- // value.
- // TODO: it would be better to let the caller to clear Subscripts, similar
- // to how we handle Sizes.
- if (Sizes.empty() || Subscripts.size() <= 1) {
- Subscripts.clear();
+ SmallVector<const SCEV *, 4> SCEVSizes;
+ const SCEV *ElementSize = SE->getElementSize(Inst);
+ if (!delinearizeUsingArrayInfo(*SE, AccessFn, Subscripts, SCEVSizes,
+ ElementSize))
return false;
- }
-
- // Check that for identical base pointers we do not miss index offsets
- // that have been added before this GEP is applied.
- Value *SrcBasePtr = SrcGEP->getOperand(0)->stripPointerCasts();
- const SCEVUnknown *SrcBase =
- dyn...
[truncated]
|
|
@llvm/pr-subscribers-llvm-transforms Author: Sebastian Pop (sebpop) ChangesThis patch series removes the use of GEP instructions from delinearization.
Testcases adjusted to maintain the patterns they test for. The third patch can be rewritten (if alloca's and globals are also going to be scheduled for removal from LLVM IR) to send the data from the front-ends attached to The current patch does not change the LLVM IR, it assumes the array info can be discovered through inter-procedural analysis, and if IPA is not available, it relies on the other symbolic delinearization techniques. Patch is 225.80 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/156215.diff 93 Files Affected:
diff --git a/llvm/include/llvm/Analysis/Delinearization.h b/llvm/include/llvm/Analysis/Delinearization.h
index 434cfb61699d6..2eda019abaf46 100644
--- a/llvm/include/llvm/Analysis/Delinearization.h
+++ b/llvm/include/llvm/Analysis/Delinearization.h
@@ -112,6 +112,13 @@ void delinearize(ScalarEvolution &SE, const SCEV *Expr,
SmallVectorImpl<const SCEV *> &Subscripts,
SmallVectorImpl<const SCEV *> &Sizes, const SCEV *ElementSize);
+/// Same as delinearize. TODO: only use delinearize() instead of other internal
+/// functions.
+bool delinearizeUsingArrayInfo(ScalarEvolution &SE, const SCEV *AccessFn,
+ SmallVectorImpl<const SCEV *> &Subscripts,
+ SmallVectorImpl<const SCEV *> &Sizes,
+ const SCEV *ElementSize);
+
/// Compute the dimensions of fixed size array from \Expr and save the results
/// in \p Sizes.
bool findFixedSizeArrayDimensions(ScalarEvolution &SE, const SCEV *Expr,
@@ -132,28 +139,21 @@ bool findFixedSizeArrayDimensions(ScalarEvolution &SE, const SCEV *Expr,
/// Different from the normal delinearization, this function assumes that NO
/// terms exist in the \p Expr. In other words, it assumes that the all step
/// values are constant.
-///
-/// This function is intended to replace getIndexExpressionsFromGEP and
-/// tryDelinearizeFixedSizeImpl. They rely on the GEP source element type so
-/// that they will be removed in the future.
bool delinearizeFixedSizeArray(ScalarEvolution &SE, const SCEV *Expr,
SmallVectorImpl<const SCEV *> &Subscripts,
SmallVectorImpl<const SCEV *> &Sizes,
const SCEV *ElementSize);
-/// Gathers the individual index expressions from a GEP instruction.
-///
-/// This function optimistically assumes the GEP references into a fixed size
-/// array. If this is actually true, this function returns a list of array
-/// subscript expressions in \p Subscripts and a list of integers describing
-/// the size of the individual array dimensions in \p Sizes. Both lists have
-/// either equal length or the size list is one element shorter in case there
-/// is no known size available for the outermost array dimension. Returns true
-/// if successful and false otherwise.
-bool getIndexExpressionsFromGEP(ScalarEvolution &SE,
- const GetElementPtrInst *GEP,
- SmallVectorImpl<const SCEV *> &Subscripts,
- SmallVectorImpl<int> &Sizes);
+/// Compute access functions for each subscript in a delinearized array access.
+void computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
+ SmallVectorImpl<const SCEV *> &Subscripts,
+ SmallVectorImpl<const SCEV *> &Sizes,
+ const SCEV *ElementSize);
+
+/// Backward compatibility wrapper for computeAccessFunctions.
+void computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
+ SmallVectorImpl<const SCEV *> &Subscripts,
+ SmallVectorImpl<const SCEV *> &Sizes);
/// Implementation of fixed size array delinearization. Try to delinearize
/// access function for a fixed size multi-dimensional array, by deriving
diff --git a/llvm/lib/Analysis/Delinearization.cpp b/llvm/lib/Analysis/Delinearization.cpp
index 762d9191aab1e..873060d5f7f2a 100644
--- a/llvm/lib/Analysis/Delinearization.cpp
+++ b/llvm/lib/Analysis/Delinearization.cpp
@@ -14,15 +14,18 @@
//===----------------------------------------------------------------------===//
#include "llvm/Analysis/Delinearization.h"
+#include "llvm/ADT/DenseMap.h"
#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/ScalarEvolution.h"
#include "llvm/Analysis/ScalarEvolutionDivision.h"
#include "llvm/Analysis/ScalarEvolutionExpressions.h"
#include "llvm/IR/Constants.h"
+#include "llvm/IR/DataLayout.h"
#include "llvm/IR/DerivedTypes.h"
#include "llvm/IR/Function.h"
#include "llvm/IR/InstIterator.h"
#include "llvm/IR/Instructions.h"
+#include "llvm/IR/Module.h"
#include "llvm/IR/PassManager.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Debug.h"
@@ -182,7 +185,7 @@ void llvm::collectParametricTerms(ScalarEvolution &SE, const SCEV *Expr,
LLVM_DEBUG({
dbgs() << "Strides:\n";
for (const SCEV *S : Strides)
- dbgs() << *S << "\n";
+ dbgs() << " " << *S << "\n";
});
for (const SCEV *S : Strides) {
@@ -193,7 +196,7 @@ void llvm::collectParametricTerms(ScalarEvolution &SE, const SCEV *Expr,
LLVM_DEBUG({
dbgs() << "Terms:\n";
for (const SCEV *T : Terms)
- dbgs() << *T << "\n";
+ dbgs() << " " << *T << "\n";
});
SCEVCollectAddRecMultiplies MulCollector(Terms, SE);
@@ -294,7 +297,7 @@ void llvm::findArrayDimensions(ScalarEvolution &SE,
LLVM_DEBUG({
dbgs() << "Terms:\n";
for (const SCEV *T : Terms)
- dbgs() << *T << "\n";
+ dbgs() << " " << *T << "\n";
});
// Remove duplicates.
@@ -325,7 +328,7 @@ void llvm::findArrayDimensions(ScalarEvolution &SE,
LLVM_DEBUG({
dbgs() << "Terms after sorting:\n";
for (const SCEV *T : NewTerms)
- dbgs() << *T << "\n";
+ dbgs() << " " << *T << "\n";
});
if (NewTerms.empty() || !findArrayDimensionsRec(SE, NewTerms, Sizes)) {
@@ -339,13 +342,14 @@ void llvm::findArrayDimensions(ScalarEvolution &SE,
LLVM_DEBUG({
dbgs() << "Sizes:\n";
for (const SCEV *S : Sizes)
- dbgs() << *S << "\n";
+ dbgs() << " " << *S << "\n";
});
}
void llvm::computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
SmallVectorImpl<const SCEV *> &Subscripts,
- SmallVectorImpl<const SCEV *> &Sizes) {
+ SmallVectorImpl<const SCEV *> &Sizes,
+ const SCEV *ElementSize) {
// Early exit in case this SCEV is not an affine multivariate function.
if (Sizes.empty())
return;
@@ -354,20 +358,32 @@ void llvm::computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
if (!AR->isAffine())
return;
+ if (ElementSize->isZero())
+ return;
+
+ // Clear output vector.
+ Subscripts.clear();
+
+ LLVM_DEBUG(dbgs() << "\ncomputeAccessFunctions\n"
+ << "Linearized Memory Access Function: " << *Expr << "\n");
+
const SCEV *Res = Expr;
int Last = Sizes.size() - 1;
+
for (int i = Last; i >= 0; i--) {
- const SCEV *Q, *R;
- SCEVDivision::divide(SE, Res, Sizes[i], &Q, &R);
+ const SCEV *Size = Sizes[i];
+ if (Size->isZero())
+ continue;
+ const SCEV *Q, *R;
+ SCEVDivision::divide(SE, Res, Size, &Q, &R);
LLVM_DEBUG({
- dbgs() << "Res: " << *Res << "\n";
- dbgs() << "Sizes[i]: " << *Sizes[i] << "\n";
- dbgs() << "Res divided by Sizes[i]:\n";
- dbgs() << "Quotient: " << *Q << "\n";
- dbgs() << "Remainder: " << *R << "\n";
+ dbgs() << "Computing 'MemAccFn / Sizes[" << i << "]':\n";
+ dbgs() << " MemAccFn: " << *Res << "\n";
+ dbgs() << " Sizes[" << i << "]: " << *Size << "\n";
+ dbgs() << " Quotient (Leftover): " << *Q << "\n";
+ dbgs() << " Remainder (Subscript Access Function): " << *R << "\n";
});
-
Res = Q;
// Do not record the last subscript corresponding to the size of elements in
@@ -385,22 +401,162 @@ void llvm::computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
}
// Record the access function for the current subscript.
+ LLVM_DEBUG(dbgs() << "Subscripts push_back Remainder: " << *R << "\n");
Subscripts.push_back(R);
}
- // Also push in last position the remainder of the last division: it will be
- // the access function of the innermost dimension.
- Subscripts.push_back(Res);
+ // Also push in last position the quotient "Res = Q" of the last division: it
+ // will be the access function of the outermost array dimension.
+ if (!Res->isZero()) {
+ // This is only needed when the outermost array size is not known. Res = 0
+ // when the outermost array dimension is known, as for example when reading
+ // array sizes from a local or global declaration.
+ Subscripts.push_back(Res);
+ LLVM_DEBUG(dbgs() << "Subscripts push_back Res: " << *Res << "\n");
+ }
std::reverse(Subscripts.begin(), Subscripts.end());
LLVM_DEBUG({
dbgs() << "Subscripts:\n";
for (const SCEV *S : Subscripts)
- dbgs() << *S << "\n";
+ dbgs() << " " << *S << "\n";
+ dbgs() << "\n";
});
}
+/// Backward compatibility wrapper for the old 4-parameter version.
+void llvm::computeAccessFunctions(ScalarEvolution &SE, const SCEV *Expr,
+ SmallVectorImpl<const SCEV *> &Subscripts,
+ SmallVectorImpl<const SCEV *> &Sizes) {
+ // Use the element size from the last element in Sizes array (legacy behavior)
+ if (Sizes.empty()) {
+ Subscripts.clear();
+ return;
+ }
+ const SCEV *ElementSize = Sizes.back();
+ computeAccessFunctions(SE, Expr, Subscripts, Sizes, ElementSize);
+}
+
+/// Extract array dimensions from alloca or global variable declarations.
+/// Returns true if array dimensions were successfully extracted.
+static bool
+extractArrayInfoFromAllocaOrGlobal(ScalarEvolution &SE, Value *BasePtr,
+ SmallVectorImpl<const SCEV *> &Sizes,
+ const SCEV *ElementSize) {
+ // Clear output vector.
+ Sizes.clear();
+
+ LLVM_DEBUG(dbgs() << "extractArrayInfoFromAllocaOrGlobal called with BasePtr: " << *BasePtr << "\n");
+
+ // Distinguish between simple array accesses and complex pointer arithmetic.
+ // Only apply array_info extraction to direct array accesses to avoid incorrect
+ // delinearization of complex pointer arithmetic patterns.
+ if (auto *GEP = dyn_cast<GetElementPtrInst>(BasePtr)) {
+ // Check if this is a simple array access pattern: GEP [N x T]* @array, 0, idx
+ // This represents direct indexing like array[i], which should use array dimensions.
+ if (GEP->getNumIndices() == 2) {
+ auto *FirstIdx = dyn_cast<ConstantInt>(GEP->getOperand(1));
+ if (FirstIdx && FirstIdx->isZero()) {
+ // Simple array access: extract dimensions from the underlying array type
+ Value *Source = GEP->getPointerOperand()->stripPointerCasts();
+ return extractArrayInfoFromAllocaOrGlobal(SE, Source, Sizes, ElementSize);
+ }
+ }
+ // Complex GEPs like (&array[offset])[index] represent pointer arithmetic, not
+ // simple array indexing. These should be handled by parametric delinearization
+ // to preserve the linearized byte-offset semantics rather than treating them
+ // as multidimensional array accesses.
+ return false;
+ }
+
+ // Check if BasePtr is from an alloca instruction.
+ Type *ElementType = nullptr;
+ if (auto *AI = dyn_cast<AllocaInst>(BasePtr)) {
+ ElementType = AI->getAllocatedType();
+ LLVM_DEBUG(dbgs() << "Found alloca with type: " << *ElementType << "\n");
+ } else if (auto *GV = dyn_cast<GlobalVariable>(BasePtr)) {
+ ElementType = GV->getValueType();
+ LLVM_DEBUG(dbgs() << "Found global variable with type: " << *ElementType
+ << "\n");
+ } else {
+ LLVM_DEBUG(dbgs() << "No alloca or global found for base pointer\n");
+ return false;
+ }
+
+ // Extract dimensions from nested array types.
+ Type *I64Ty = Type::getInt64Ty(SE.getContext());
+
+ while (auto *ArrayTy = dyn_cast<ArrayType>(ElementType)) {
+ uint64_t Size = ArrayTy->getNumElements();
+ const SCEV *SizeSCEV = SE.getConstant(I64Ty, Size);
+ Sizes.push_back(SizeSCEV);
+ ElementType = ArrayTy->getElementType();
+ LLVM_DEBUG(dbgs() << " Found array dimension: " << Size << "\n");
+ }
+
+ if (Sizes.empty()) {
+ LLVM_DEBUG(dbgs() << "No array dimensions found in type\n");
+ return false;
+ }
+
+ // Add element size as the last element for computeAccessFunctions algorithm.
+ Sizes.push_back(ElementSize);
+
+ LLVM_DEBUG({
+ dbgs() << "Extracted array info from alloca/global for base pointer "
+ << *BasePtr << "\n";
+ dbgs() << "Dimensions: ";
+ for (const SCEV *Size : Sizes)
+ dbgs() << *Size << " ";
+ dbgs() << "\n";
+ });
+
+ return true;
+}
+
+bool llvm::delinearizeUsingArrayInfo(ScalarEvolution &SE, const SCEV *AccessFn,
+ SmallVectorImpl<const SCEV *> &Subscripts,
+ SmallVectorImpl<const SCEV *> &Sizes,
+ const SCEV *ElementSize) {
+ // Clear output vectors.
+ Subscripts.clear();
+ Sizes.clear();
+
+ const SCEVUnknown *BasePointer =
+ dyn_cast<SCEVUnknown>(SE.getPointerBase(AccessFn));
+ if (!BasePointer) {
+ LLVM_DEBUG(dbgs() << "no BasePointer for AccessFn: " << *AccessFn << "\n");
+ return false;
+ }
+
+ Value *BasePtr = BasePointer->getValue();
+
+ // Extract array dimensions from alloca or global declarations.
+ if (!extractArrayInfoFromAllocaOrGlobal(SE, BasePtr, Sizes, ElementSize))
+ return false;
+
+ // Get the full SCEV expression and subtract the base pointer to get
+ // offset-only expression.
+ const SCEV *Expr = SE.getMinusSCEV(AccessFn, BasePointer);
+
+ computeAccessFunctions(SE, Expr, Subscripts, Sizes, ElementSize);
+ if (Sizes.empty() || Subscripts.empty())
+ return false;
+
+ // Validate dimension consistency: subscripts should match array dimensions
+ // (Sizes includes element size as last element, so array dimensions = Sizes.size() - 1)
+ unsigned ArrayDims = Sizes.size() - 1;
+ if (Subscripts.size() != ArrayDims) {
+ LLVM_DEBUG(dbgs() << "delinearizeUsingArrayInfo: Dimension mismatch - "
+ << Subscripts.size() << " subscripts for " << ArrayDims
+ << " array dimensions. Falling back to parametric method.\n");
+ return false;
+ }
+
+ return true;
+}
+
/// Splits the SCEV into two vectors of SCEVs representing the subscripts and
/// sizes of an array access. Returns the remainder of the delinearization that
/// is the offset start of the array. The SCEV->delinearize algorithm computes
@@ -454,36 +610,35 @@ void llvm::delinearize(ScalarEvolution &SE, const SCEV *Expr,
SmallVectorImpl<const SCEV *> &Subscripts,
SmallVectorImpl<const SCEV *> &Sizes,
const SCEV *ElementSize) {
- // First step: collect parametric terms.
+ // Clear output vectors.
+ Subscripts.clear();
+ Sizes.clear();
+
+ // Try array_info extraction.
+ if (delinearizeUsingArrayInfo(SE, Expr, Subscripts, Sizes, ElementSize))
+ return;
+ LLVM_DEBUG(dbgs() << "delinearize falling back to parametric method\n");
+
+ // Fall back to parametric delinearization.
+ const SCEVUnknown *BasePointer =
+ dyn_cast<SCEVUnknown>(SE.getPointerBase(Expr));
+ if (BasePointer)
+ Expr = SE.getMinusSCEV(Expr, BasePointer);
+
SmallVector<const SCEV *, 4> Terms;
collectParametricTerms(SE, Expr, Terms);
if (Terms.empty())
return;
- // Second step: find subscript sizes.
+ // Find subscript sizes.
findArrayDimensions(SE, Terms, Sizes, ElementSize);
if (Sizes.empty())
return;
- // Third step: compute the access functions for each subscript.
- computeAccessFunctions(SE, Expr, Subscripts, Sizes);
-
- if (Subscripts.empty())
- return;
-
- LLVM_DEBUG({
- dbgs() << "succeeded to delinearize " << *Expr << "\n";
- dbgs() << "ArrayDecl[UnknownSize]";
- for (const SCEV *S : Sizes)
- dbgs() << "[" << *S << "]";
-
- dbgs() << "\nArrayRef";
- for (const SCEV *S : Subscripts)
- dbgs() << "[" << *S << "]";
- dbgs() << "\n";
- });
+ // Compute the access functions for each subscript.
+ computeAccessFunctions(SE, Expr, Subscripts, Sizes, ElementSize);
}
static std::optional<APInt> tryIntoAPInt(const SCEV *S) {
@@ -492,6 +647,21 @@ static std::optional<APInt> tryIntoAPInt(const SCEV *S) {
return std::nullopt;
}
+/// Convert cached SCEV sizes to int sizes for compatibility.
+/// TODO: Remove this after we remove GEP delinearization.
+static void convertSCEVSizesToIntSizes(ArrayRef<const SCEV *> SCEVSizes,
+ SmallVectorImpl<int> &Sizes) {
+ for (const SCEV *S : SCEVSizes) {
+ if (auto *Const = dyn_cast<SCEVConstant>(S)) {
+ const APInt &APVal = Const->getAPInt();
+ if (APVal.isSignedIntN(32)) {
+ int intValue = APVal.getSExtValue();
+ Sizes.push_back(intValue);
+ }
+ }
+ }
+}
+
/// Collects the absolute values of constant steps for all induction variables.
/// Returns true if we can prove that all step recurrences are constants and \p
/// Expr is divisible by \p ElementSize. Each step recurrence is stored in \p
@@ -646,6 +816,9 @@ bool llvm::delinearizeFixedSizeArray(ScalarEvolution &SE, const SCEV *Expr,
SmallVectorImpl<const SCEV *> &Subscripts,
SmallVectorImpl<const SCEV *> &Sizes,
const SCEV *ElementSize) {
+ // Clear output vectors.
+ Subscripts.clear();
+ Sizes.clear();
// First step: find the fixed array size.
SmallVector<uint64_t, 4> ConstSizes;
@@ -659,52 +832,18 @@ bool llvm::delinearizeFixedSizeArray(ScalarEvolution &SE, const SCEV *Expr,
Sizes.push_back(SE.getConstant(Expr->getType(), Size));
// Second step: compute the access functions for each subscript.
- computeAccessFunctions(SE, Expr, Subscripts, Sizes);
+ computeAccessFunctions(SE, Expr, Subscripts, Sizes, ElementSize);
return !Subscripts.empty();
}
-bool llvm::getIndexExpressionsFromGEP(ScalarEvolution &SE,
- const GetElementPtrInst *GEP,
- SmallVectorImpl<const SCEV *> &Subscripts,
- SmallVectorImpl<int> &Sizes) {
- assert(Subscripts.empty() && Sizes.empty() &&
- "Expected output lists to be empty on entry to this function.");
- assert(GEP && "getIndexExpressionsFromGEP called with a null GEP");
- Type *Ty = nullptr;
- bool DroppedFirstDim = false;
- for (unsigned i = 1; i < GEP->getNumOperands(); i++) {
- const SCEV *Expr = SE.getSCEV(GEP->getOperand(i));
- if (i == 1) {
- Ty = GEP->getSourceElementType();
- if (auto *Const = dyn_cast<SCEVConstant>(Expr))
- if (Const->getValue()->isZero()) {
- DroppedFirstDim = true;
- continue;
- }
- Subscripts.push_back(Expr);
- continue;
- }
-
- auto *ArrayTy = dyn_cast<ArrayType>(Ty);
- if (!ArrayTy) {
- Subscripts.clear();
- Sizes.clear();
- return false;
- }
-
- Subscripts.push_back(Expr);
- if (!(DroppedFirstDim && i == 2))
- Sizes.push_back(ArrayTy->getNumElements());
-
- Ty = ArrayTy->getElementType();
- }
- return !Subscripts.empty();
-}
-
bool llvm::tryDelinearizeFixedSizeImpl(
ScalarEvolution *SE, Instruction *Inst, const SCEV *AccessFn,
SmallVectorImpl<const SCEV *> &Subscripts, SmallVectorImpl<int> &Sizes) {
+ // Clear output vectors.
+ Subscripts.clear();
+ Sizes.clear();
+
Value *SrcPtr = getLoadStorePointerOperand(Inst);
// Check the simple case where the array dimensions are fixed size.
@@ -712,31 +851,17 @@ bool llvm::tryDelinearizeFixedSizeImpl(
if (!SrcGEP)
return false;
- getIndexExpressionsFromGEP(*SE, SrcGEP, Subscripts, Sizes);
-
- // Check that the two size arrays are non-empty and equal in length and
- // value.
- // TODO: it would be better to let the caller to clear Subscripts, similar
- // to how we handle Sizes.
- if (Sizes.empty() || Subscripts.size() <= 1) {
- Subscripts.clear();
+ SmallVector<const SCEV *, 4> SCEVSizes;
+ const SCEV *ElementSize = SE->getElementSize(Inst);
+ if (!delinearizeUsingArrayInfo(*SE, AccessFn, Subscripts, SCEVSizes,
+ ElementSize))
return false;
- }
-
- // Check that for identical base pointers we do not miss index offsets
- // that have been added before this GEP is applied.
- Value *SrcBasePtr = SrcGEP->getOperand(0)->stripPointerCasts();
- const SCEVUnknown *SrcBase =
- dyn...
[truncated]
|
…tions This implements an alternative to extract multi-dimensional array information directly from alloca instructions and global variable declarations instead of relying on parsing the information from GEP instructions. The functionality for GEP delinearization remains in this patch such that we can evaluate regressions of the array_info extraction. Next patch removes the code for GEP delinearization.
This finishes removing the code of GEP delinearization. Follow up patches will clean the interface, in particular Sizes array of ints will be rewritten as an array of SCEVs to avoid unnecessary translations. Another painful point to be addressed in subsequent patches is unifying the delinearize() interface to avoid users directly hit in the internals of delinearize. The many uses of delinearization internals makes changes to the core delinearization algorithm hard to modify.
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
sebpop
force-pushed
the
delin-GEP-removal
branch
from
6b1020d to
1f39c61
Compare
|
It's good to see the flow from beginning to end, but I think we want to comment on the patches individually and merge them separately, so I think each change needs its own merge request. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch series removes the use of GEP instructions from delinearization.
Testcases adjusted to maintain the patterns they test for.
The third patch can be rewritten (if alloca's and globals are also going to be scheduled for removal from LLVM IR) to send the data from the front-ends attached to
assume(true)statements: proof of concept to send data from clang / flang with assumes is still incomplete as it would only send the equivalent of the alloca info, still needs to send info about global vars. This approach would also need some agreement to attach the array_info to the assumes, which changes the LLVM IR.The current patch does not change the LLVM IR, it assumes the array info can be discovered through inter-procedural analysis, and if IPA is not available, it relies on the other symbolic delinearization techniques.