[RISCV] Add TSFlag for reading past VL behaviour. NFCI #149704

lukel97 · 2025-07-20T12:10:07Z

Currently we have a switch statement that checks if a vector instruction may read elements past VL. However it currently doesn't account for instructions in vendor extensions.

Handling all possible vendor instructions will result in quite a lot of opcodes being added, so I've created a new TSFlag that we can declare in TableGen, and added it to the existing instruction definitions.

I've tried to be conservative as possible here: All SiFive vendor vector instructions should be covered by the flag, as well as all of XRivosVizip, and ri.vextract from XRivosVisni.

For now this should be NFC because coincidentally, these instructions aren't handled in getOperandInfo, so RISCVVLOptimizer should currently avoid touching them despite them being liberally handled in getMinimumVLForUser.

However in an upcoming patch we'll need to also bail in getMinimumVLForUser, so this prepares for it.

llvmbot · 2025-07-20T12:10:37Z

@llvm/pr-subscribers-backend-risc-v

Author: Luke Lau (lukel97)

Changes

Stacked on #149698

Currently we have a switch statement that checks if a vector instruction may read elements past VL. However it currently doesn't account for instructions in vendor extensions.

Handling all possible vendor instructions will result in quite a lot of opcodes being added, so I've created a new TSFlag that we can declare in TableGen, and added it to the existing instruction definitions.

I've tried to be conservative as possible here: All SiFive vendor vector instructions should be covered by the flag, as well as all of XRivosVizip, and ri.vextract from XRivosVisni.

For now this should be NFC because coincidentally, these instructions aren't handled in getOperandInfo, so RISCVVLOptimizer should currently avoid touching them despite them being liberally handled in getMinimumVLForUser.

However in an upcoming patch we'll need to also bail in getMinimumVLForUser, so this prepares for it.

Full diff: https://github.com/llvm/llvm-project/pull/149704.diff

7 Files Affected:

(modified) llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h (+9)
(modified) llvm/lib/Target/RISCV/RISCVInstrFormats.td (+7)
(modified) llvm/lib/Target/RISCV/RISCVInstrInfoV.td (+7-6)
(modified) llvm/lib/Target/RISCV/RISCVInstrInfoXRivos.td (+2-1)
(modified) llvm/lib/Target/RISCV/RISCVInstrInfoXSf.td (+2)
(modified) llvm/lib/Target/RISCV/RISCVInstrInfoXSfmm.td (+5)
(modified) llvm/lib/Target/RISCV/RISCVVLOptimizer.cpp (+7-30)

diff --git a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h
index 4c8dcf376755b..210051c5ccd3f 100644
--- a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h
+++ b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVBaseInfo.h
@@ -139,6 +139,9 @@ enum {
   // 3 -> SEW * 4
   DestEEWShift = ElementsDependOnMaskShift + 1,
   DestEEWMask = 3ULL << DestEEWShift,
+
+  ReadsPastVLShift = DestEEWShift + 2,
+  ReadsPastVLMask = 1ULL << ReadsPastVLShift,
 };
 
 // Helper functions to read TSFlags.
@@ -195,6 +198,12 @@ static inline bool elementsDependOnMask(uint64_t TSFlags) {
   return TSFlags & ElementsDependOnMaskMask;
 }
 
+/// \returns true if the instruction may read elements past VL, e.g.
+/// vslidedown/vrgather
+static inline bool readsPastVL(uint64_t TSFlags) {
+  return TSFlags & ReadsPastVLMask;
+}
+
 static inline unsigned getVLOpNum(const MCInstrDesc &Desc) {
   const uint64_t TSFlags = Desc.TSFlags;
   // This method is only called if we expect to have a VL operand, and all
diff --git a/llvm/lib/Target/RISCV/RISCVInstrFormats.td b/llvm/lib/Target/RISCV/RISCVInstrFormats.td
index b6b64b57b1b3e..2e689924af327 100644
--- a/llvm/lib/Target/RISCV/RISCVInstrFormats.td
+++ b/llvm/lib/Target/RISCV/RISCVInstrFormats.td
@@ -174,6 +174,7 @@ class EltDeps<bit vl, bit mask> {
 
 def EltDepsNone      : EltDeps<vl=0, mask=0>;
 def EltDepsVL        : EltDeps<vl=1, mask=0>;
+def EltDepsMask      : EltDeps<vl=0, mask=1>;
 def EltDepsVLMask    : EltDeps<vl=1, mask=1>;
 
 class EEW <bits<2> val> {
@@ -258,6 +259,12 @@ class RVInstCommon<dag outs, dag ins, string opcodestr, string argstr,
   // Indicates the EEW of a vector instruction's destination operand.
   EEW DestEEW = EEWSEWx1;
   let TSFlags{25-24} = DestEEW.Value;
+
+  // Some vector instructions like vslidedown/vrgather will read elements past
+  // VL, and should be marked to make sure RISCVVLOptimizer doesn't reduce its
+  // operands' VLs.
+  bit ReadsPastVL = 0;
+  let TSFlags{26} = ReadsPastVL;
 }
 
 class RVInst<dag outs, dag ins, string opcodestr, string argstr,
diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfoV.td b/llvm/lib/Target/RISCV/RISCVInstrInfoV.td
index 5d13a877c90a2..cebab2112d02d 100644
--- a/llvm/lib/Target/RISCV/RISCVInstrInfoV.td
+++ b/llvm/lib/Target/RISCV/RISCVInstrInfoV.td
@@ -1642,7 +1642,7 @@ def VFIRST_M : RVInstV<0b010000, 0b10001, OPMVV, (outs GPR:$vd),
 
 def : MnemonicAlias<"vpopc.m", "vcpop.m">;
 
-let Constraints = "@earlyclobber $vd", RVVConstraint = Iota, ElementsDependOn = EltDepsVLMask in {
+let Constraints = "@earlyclobber $vd", RVVConstraint = Iota, ElementsDependOn = EltDepsMask in {
 
 let DestEEW = EEW1 in {
 // vmsbf.m set-before-first mask bit
@@ -1655,7 +1655,7 @@ defm VMSOF_M : VMSFS_MV_V<"vmsof.m", 0b010100, 0b00010>;
 // Vector Iota Instruction
 defm VIOTA_M : VIOTA_MV_V<"viota.m", 0b010100, 0b10000>;
 
-} // Constraints = "@earlyclobber $vd", RVVConstraint = Iota, ElementsDependOn = EltDepsVLMask
+} // Constraints = "@earlyclobber $vd", RVVConstraint = Iota, ElementsDependOn = EltDepsMask
 
 // Vector Element Index Instruction
 let hasSideEffects = 0, mayLoad = 0, mayStore = 0 in {
@@ -1703,8 +1703,9 @@ let Constraints = "@earlyclobber $vd", RVVConstraint = SlideUp in {
 defm VSLIDEUP_V : VSLD_IV_X_I<"vslideup", 0b001110, /*slidesUp=*/true>;
 defm VSLIDE1UP_V : VSLD1_MV_X<"vslide1up", 0b001110>;
 } // Constraints = "@earlyclobber $vd", RVVConstraint = SlideUp
+let ReadsPastVL = 1 in
 defm VSLIDEDOWN_V : VSLD_IV_X_I<"vslidedown", 0b001111, /*slidesUp=*/false>;
-let ElementsDependOn = EltDepsVL in
+let ElementsDependOn = EltDepsVL, ReadsPastVL = 1 in
 defm VSLIDE1DOWN_V : VSLD1_MV_X<"vslide1down", 0b001111>;
 } // Predicates = [HasVInstructions]
 
@@ -1712,19 +1713,19 @@ let Predicates = [HasVInstructionsAnyF] in {
 let Constraints = "@earlyclobber $vd", RVVConstraint = SlideUp in {
 defm VFSLIDE1UP_V : VSLD1_FV_F<"vfslide1up", 0b001110>;
 } // Constraints = "@earlyclobber $vd", RVVConstraint = SlideUp
-let ElementsDependOn = EltDepsVL in
+let ElementsDependOn = EltDepsVL, ReadsPastVL = 1 in
 defm VFSLIDE1DOWN_V : VSLD1_FV_F<"vfslide1down", 0b001111>;
 } // Predicates = [HasVInstructionsAnyF]
 
 let Predicates = [HasVInstructions] in {
 // Vector Register Gather Instruction
-let Constraints = "@earlyclobber $vd", RVVConstraint = Vrgather in {
+let Constraints = "@earlyclobber $vd", RVVConstraint = Vrgather, ReadsPastVL = 1 in {
 defm VRGATHER_V : VGTR_IV_V_X_I<"vrgather", 0b001100>;
 def VRGATHEREI16_VV : VALUVV<0b001110, OPIVV, "vrgatherei16.vv">,
                       SchedBinaryMC<"WriteVRGatherEI16VV",
                                     "ReadVRGatherEI16VV_data",
                                     "ReadVRGatherEI16VV_index">;
-} // Constraints = "@earlyclobber $vd", RVVConstraint = Vrgather
+} // Constraints = "@earlyclobber $vd", RVVConstraint = Vrgather, ReadsPastVL = 1
 
 // Vector Compress Instruction
 let Constraints = "@earlyclobber $vd", RVVConstraint = Vcompress, ElementsDependOn = EltDepsVLMask in {
diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfoXRivos.td b/llvm/lib/Target/RISCV/RISCVInstrInfoXRivos.td
index 3912eb0d16c59..6e8e910249c82 100644
--- a/llvm/lib/Target/RISCV/RISCVInstrInfoXRivos.td
+++ b/llvm/lib/Target/RISCV/RISCVInstrInfoXRivos.td
@@ -58,7 +58,7 @@ class CustomRivosXVI<bits<6> funct6, RISCVVFormat opv, dag outs, dag ins,
 
 let Predicates = [HasVendorXRivosVizip], DecoderNamespace = "XRivos",
   Constraints = "@earlyclobber $vd", RVVConstraint = Vrgather,
-  Inst<6-0> = OPC_CUSTOM_2.Value in  {
+  Inst<6-0> = OPC_CUSTOM_2.Value, ReadsPastVL = 1 in  {
 defm RI_VZIPEVEN_V : VALU_IV_V<"ri.vzipeven", 0b001100>;
 defm RI_VZIPODD_V : VALU_IV_V<"ri.vzipodd", 0b011100>;
 defm RI_VZIP2A_V : VALU_IV_V<"ri.vzip2a", 0b000100>;
@@ -126,6 +126,7 @@ def RI_VINSERT : CustomRivosVXI<0b010000, OPMVX, (outs VR:$vd_wb),
                                 (ins VR:$vd, GPR:$rs1, uimm5:$imm),
                                 "ri.vinsert.v.x", "$vd, $rs1, $imm">;
 
+let ReadsPastVL = 1 in
 def RI_VEXTRACT : CustomRivosXVI<0b010111, OPMVV, (outs GPR:$rd),
                                 (ins VR:$vs2, uimm5:$imm),
                                 "ri.vextract.x.v", "$rd, $vs2, $imm">;
diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfoXSf.td b/llvm/lib/Target/RISCV/RISCVInstrInfoXSf.td
index 17fb75eb851c4..8b3b3ed6db188 100644
--- a/llvm/lib/Target/RISCV/RISCVInstrInfoXSf.td
+++ b/llvm/lib/Target/RISCV/RISCVInstrInfoXSf.td
@@ -74,6 +74,7 @@ class RVInstVCCustom2<bits<4> funct6_hi4, bits<3> funct3, dag outs, dag ins,
   let Uses = [VL, VTYPE];
   let RVVConstraint = NoConstraint;
   let ElementsDependOn = EltDepsVLMask;
+  let ReadsPastVL = 1;
 }
 
 class RVInstVCFCustom2<bits<4> funct6_hi4, bits<3> funct3, dag outs, dag ins,
@@ -98,6 +99,7 @@ class RVInstVCFCustom2<bits<4> funct6_hi4, bits<3> funct3, dag outs, dag ins,
   let Uses = [VL, VTYPE];
   let RVVConstraint = NoConstraint;
   let ElementsDependOn = EltDepsVLMask;
+  let ReadsPastVL = 1;
 }
 
 class VCIXInfo<string suffix, VCIXType type, DAGOperand TyRd,
diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfoXSfmm.td b/llvm/lib/Target/RISCV/RISCVInstrInfoXSfmm.td
index 66cb2d53da960..a5ee701386b6d 100644
--- a/llvm/lib/Target/RISCV/RISCVInstrInfoXSfmm.td
+++ b/llvm/lib/Target/RISCV/RISCVInstrInfoXSfmm.td
@@ -65,6 +65,7 @@ class SFInstTileMemOp<dag outs, dag ins, bits<3> nf, RISCVOpcode opcode,
   let Inst{6-0} = opcode.Value;
 
   let Uses = [VTYPE, VL];
+  let ReadsPastVL = 1;
 }
 
 let hasSideEffects = 0, mayLoad = 1, mayStore = 0 in
@@ -94,6 +95,7 @@ class SFInstTileMoveOp<bits<6> funct6, dag outs, dag ins, string opcodestr,
   let Inst{6-0} = OPC_OP_V.Value;
 
   let Uses = [VTYPE, VL];
+  let ReadsPastVL = 1;
 }
 
 let hasSideEffects = 0, mayLoad = 0, mayStore = 0 in
@@ -113,6 +115,7 @@ class SFInstMatmulF<dag outs, dag ins, string opcodestr, string argstr>
   let Inst{6-0} = OPC_OP_VE.Value;
 
   let Uses = [VTYPE, VL];
+  let ReadsPastVL = 1;
 }
 
 let hasSideEffects = 0, mayLoad = 0, mayStore = 0 in
@@ -135,6 +138,7 @@ class SFInstMatmulF8<bit a, bit b, dag outs, dag ins,
   let Inst{6-0} = OPC_OP_VE.Value;
 
   let Uses = [VTYPE, VL];
+  let ReadsPastVL = 1;
 }
 
 
@@ -167,6 +171,7 @@ class SFInstMatmulI8<bit funct6_1, bit a, bit b, dag outs, dag ins,
   let Inst{6-0} = OPC_OP_VE.Value;
 
   let Uses = [VTYPE, VL];
+  let ReadsPastVL = 1;
 }
 
 class I8Encode<bit encoding, string name> {
diff --git a/llvm/lib/Target/RISCV/RISCVVLOptimizer.cpp b/llvm/lib/Target/RISCV/RISCVVLOptimizer.cpp
index 074a776405d6a..ad0445ce808f8 100644
--- a/llvm/lib/Target/RISCV/RISCVVLOptimizer.cpp
+++ b/llvm/lib/Target/RISCV/RISCVVLOptimizer.cpp
@@ -33,6 +33,7 @@ namespace {
 class RISCVVLOptimizer : public MachineFunctionPass {
   const MachineRegisterInfo *MRI;
   const MachineDominatorTree *MDT;
+  const TargetInstrInfo *TII;
 
 public:
   static char ID;
@@ -1204,34 +1205,6 @@ static bool isVectorOpUsedAsScalarOp(const MachineOperand &MO) {
   }
 }
 
-/// Return true if MI may read elements past VL.
-static bool mayReadPastVL(const MachineInstr &MI) {
-  const RISCVVPseudosTable::PseudoInfo *RVV =
-      RISCVVPseudosTable::getPseudoInfo(MI.getOpcode());
-  if (!RVV)
-    return true;
-
-  switch (RVV->BaseInstr) {
-  // vslidedown instructions may read elements past VL. They are handled
-  // according to current tail policy.
-  case RISCV::VSLIDEDOWN_VI:
-  case RISCV::VSLIDEDOWN_VX:
-  case RISCV::VSLIDE1DOWN_VX:
-  case RISCV::VFSLIDE1DOWN_VF:
-
-  // vrgather instructions may read the source vector at any index < VLMAX,
-  // regardless of VL.
-  case RISCV::VRGATHER_VI:
-  case RISCV::VRGATHER_VV:
-  case RISCV::VRGATHER_VX:
-  case RISCV::VRGATHEREI16_VV:
-    return true;
-
-  default:
-    return false;
-  }
-}
-
 bool RISCVVLOptimizer::isCandidate(const MachineInstr &MI) const {
   const MCInstrDesc &Desc = MI.getDesc();
   if (!RISCVII::hasVLOp(Desc.TSFlags) || !RISCVII::hasSEWOp(Desc.TSFlags))
@@ -1269,7 +1242,8 @@ bool RISCVVLOptimizer::isCandidate(const MachineInstr &MI) const {
     return false;
   }
 
-  assert(!RISCVII::elementsDependOnVL(RISCV::getRVVMCOpcode(MI.getOpcode())) &&
+  assert(!RISCVII::elementsDependOnVL(
+             TII->get(RISCV::getRVVMCOpcode(MI.getOpcode())).TSFlags) &&
          "Instruction shouldn't be supported if elements depend on VL");
 
   assert(MI.getOperand(0).isReg() &&
@@ -1291,7 +1265,8 @@ RISCVVLOptimizer::getMinimumVLForUser(const MachineOperand &UserOp) const {
     return std::nullopt;
   }
 
-  if (mayReadPastVL(UserMI)) {
+  if (RISCVII::readsPastVL(
+          TII->get(RISCV::getRVVMCOpcode(UserMI.getOpcode())).TSFlags)) {
     LLVM_DEBUG(dbgs() << "    Abort because used by unsafe instruction\n");
     return std::nullopt;
   }
@@ -1473,6 +1448,8 @@ bool RISCVVLOptimizer::runOnMachineFunction(MachineFunction &MF) {
   if (!ST.hasVInstructions())
     return false;
 
+  TII = ST.getInstrInfo();
+
   // For each instruction that defines a vector, compute what VL its
   // downstream users demand.
   for (MachineBasicBlock *MBB : post_order(&MF)) {

Currently we have a switch statement that checks if a vector instruction may read elements past VL. However it currently doesn't account for instructions in vendor extensions. Handling all possible vendor instructions will result in quite a lot of opcodes being added, so I've created a new TSFlag that we can declare in TableGen, and added it to the existing instruction definitions. I've tried to be conservative as possible here: All SiFive vendor vector instructions should be covered by the flag, as well as all of XRivosVizip, and ri.vextract from XRivosVisni. For now this should be NFC because coincidentally, these instructions aren't handled in getOperandInfo, so RISCVVLOptimizer should currently avoid touching them despite them being liberally handled in getMinimumVLForUser. However in an upcoming patch we'll need to also bail in getMinimumVLForUser, so this prepares for it.

llvm/lib/Target/RISCV/RISCVVLOptimizer.cpp

kito-cheng · 2025-07-21T14:25:59Z

I like the direction for moving info into .td, I've consider moving getOperandLog2EEW/isSupportedInstr into td file once...but it seems non-trivial work...:P

lukel97 · 2025-07-21T16:18:49Z

I like the direction for moving info into .td, I've consider moving getOperandLog2EEW/isSupportedInstr into td file once...but it seems non-trivial work...:P

For isSupportedInstr there is #149018, and as for getOperandLog2EEW I have a rough patch sitting around locally that kinda moves it into tablegen. The idea is that there's only a fixed number of "EEW formats" for each instruction, e.g. EEW=SEW, EEW=SEW*2, etc. So in tablegen there is an enum that contains each format, and each instruction has their format encoded in TSFlags

preames

The generic and Rivos custom look fine. I'd like to have someone from SiFive approve their custom instructions.

preames · 2025-07-23T17:24:18Z

llvm/lib/Target/RISCV/RISCVInstrInfoXRivos.td

 let Predicates = [HasVendorXRivosVizip], DecoderNamespace = "XRivos",
  Constraints = "@earlyclobber $vd", RVVConstraint = Vrgather,
-  Inst<6-0> = OPC_CUSTOM_2.Value in  {
+  Inst<6-0> = OPC_CUSTOM_2.Value, ReadsPastVL = 1 in  {


Not all of these can read past VL, but I'll fix this up in a following commit after I re-read the specification closely.

lukel97 · 2025-07-30T08:14:40Z

Ping

preames · 2025-07-30T22:42:16Z

Ping

@topperc @mshockwave Any comments on the sifive custom instructions?

topperc · 2025-07-30T23:27:20Z

I've tried to be conservative as possible here: All SiFive vendor vector instructions should be covered by the flag, as well as all of XRivosVizip, and ri.vextract from XRivosVisni.

I think sf.vfnrclip.x.f.qf and sf.vfnrclip.xu.f.qf were missed, but I don't think they read past VL.

The SiFive vmacc instructions require VL to be a multiple of 16 or 32 and operate in blocks. So ReadsPastVL is a good idea for those.

The VCIX instructions allow custom implementations so probably best to not change the VL for those.

lukel97 · 2025-07-31T02:43:57Z

The SiFive vmacc instructions require VL to be a multiple of 16 or 32 and operate in blocks. So ReadsPastVL is a good idea for those.

The VCIX instructions allow custom implementations so probably best to not change the VL for those.

I think the vmacc/VCIX/VCFX instructions are all based off of RVInstVC[F]Custom2 which should have ReadsPastVL set

lukel97 · 2025-08-14T07:41:06Z

Ping

preames

My prior LGTM still stands.

topperc

LGTM

lukel97 requested review from BeMg, michaelmaitland, mikhailramalho, mshockwave, preames, topperc and wangpc-pp July 20, 2025 12:10

llvmbot added the backend:RISC-V label Jul 20, 2025

lukel97 force-pushed the vloptimizer/reads-past-vl branch from 5e0c0a3 to fc44f43 Compare July 21, 2025 06:52

wangpc-pp reviewed Jul 21, 2025

View reviewed changes

llvm/lib/Target/RISCV/RISCVVLOptimizer.cpp Show resolved Hide resolved

preames reviewed Jul 23, 2025

View reviewed changes

Merge branch 'main' into vloptimizer/reads-past-vl

86a6ff0

lukel97 mentioned this pull request Jul 30, 2025

[RISCV] Handle recurrences in RISCVVLOptimizer #151285

Merged

preames approved these changes Aug 14, 2025

View reviewed changes

topperc approved these changes Aug 14, 2025

View reviewed changes

Merge branch 'main' into vloptimizer/reads-past-vl

bfa0c0b

lukel97 enabled auto-merge (squash) August 15, 2025 00:45

lukel97 merged commit e261f28 into llvm:main Aug 15, 2025
9 checks passed

[RISCV] Add TSFlag for reading past VL behaviour. NFCI #149704

[RISCV] Add TSFlag for reading past VL behaviour. NFCI #149704

Uh oh!

Conversation

lukel97 commented Jul 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented Jul 20, 2025

Uh oh!

Uh oh!

kito-cheng commented Jul 21, 2025

Uh oh!

lukel97 commented Jul 21, 2025

Uh oh!

preames left a comment

Choose a reason for hiding this comment

Uh oh!

preames Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

lukel97 commented Jul 30, 2025

Uh oh!

preames commented Jul 30, 2025

Uh oh!

topperc commented Jul 30, 2025

Uh oh!

lukel97 commented Jul 31, 2025

Uh oh!

lukel97 commented Aug 14, 2025

Uh oh!

preames left a comment

Choose a reason for hiding this comment

Uh oh!

topperc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

lukel97 commented Jul 20, 2025 •

edited

Loading