[RISCV] Implement multi-lib reuse rule for RISC-V bare-metal toolchain #73765

4vtomat · 2023-11-29T08:43:20Z

Extend the multi-lib re-use selection mechanism for RISC-V.
This funciton will try to re-use multi-lib if they are compatible.
Definition of compatible:

ABI must be the same.
multi-lib is a subset of current arch, e.g. multi-lib=march=rv32im
is a subset of march=rv32imc.
march that contains atomic extension can't reuse multi-lib that
doesn't has atomic, vice versa. e.g. multi-lib=march=rv32im and
march=rv32ima are not compatible, because software and hardware
atomic operation can't work together correctly.

llvmbot · 2023-11-29T08:43:49Z

@llvm/pr-subscribers-clang

@llvm/pr-subscribers-clang-driver

Author: Brandon Wu (4vtomat)

Changes

Extend the multi-lib re-use selection mechanism for RISC-V.
This funciton will try to re-use multi-lib if they are compatible.
Definition of compatible:

ABI must be the same.
multi-lib is a subset of current arch, e.g. multi-lib=march=rv32im
is a subset of march=rv32imc.
march that contains atomic extension can't reuse multi-lib that
doesn't has atomic, vice versa. e.g. multi-lib=march=rv32im and
march=rv32ima are not compatible, because software and hardware
atomic operation can't work together correctly.

Full diff: https://github.com/llvm/llvm-project/pull/73765.diff

6 Files Affected:

(modified) clang/lib/Driver/ToolChains/Gnu.cpp (+130-4)
(added) clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ic/ilp32/crtbegin.o ()
(added) clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32if/ilp32f/crtbegin.o ()
(added) clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ifc/ilp32/crtbegin.o ()
(added) clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ifc/ilp32f/crtbegin.o ()
(added) clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c (+86)

diff --git a/clang/lib/Driver/ToolChains/Gnu.cpp b/clang/lib/Driver/ToolChains/Gnu.cpp
index 16cf5707227264d..ffc53377d97bd02 100644
--- a/clang/lib/Driver/ToolChains/Gnu.cpp
+++ b/clang/lib/Driver/ToolChains/Gnu.cpp
@@ -30,6 +30,7 @@
 #include "llvm/Option/ArgList.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/RISCVISAInfo.h"
 #include "llvm/Support/VirtualFileSystem.h"
 #include "llvm/TargetParser/TargetParser.h"
 #include <system_error>
@@ -1715,6 +1716,129 @@ static void findCSKYMultilibs(const Driver &D, const llvm::Triple &TargetTriple,
     Result.Multilibs = CSKYMultilibs;
 }
 
+/// Extend the multi-lib re-use selection mechanism for RISC-V.
+/// This funciton will try to re-use multi-lib if they are compatible.
+/// Definition of compatible:
+///   - ABI must be the same.
+///   - multi-lib is a subset of current arch, e.g. multi-lib=march=rv32im
+///     is a subset of march=rv32imc.
+///   - march that contains atomic extension can't reuse multi-lib that
+///     doesn't has atomic, vice versa. e.g. multi-lib=march=rv32im and
+///     march=rv32ima are not compatible, because software and hardware
+///     atomic operation can't work together correctly.
+static bool
+RISCVMultilibSelect(const MultilibSet &RISCVMultilibSet, StringRef Arch,
+                    const Multilib::flags_list &Flags,
+                    llvm::SmallVector<Multilib> &SelectedMultilibs) {
+  // Try to find the perfect matching multi-lib first.
+  if (RISCVMultilibSet.select(Flags, SelectedMultilibs))
+    return true;
+
+  llvm::StringMap<bool> FlagSet;
+  Multilib::flags_list NewFlags;
+  std::vector<MultilibBuilder> NewMultilibs;
+
+  auto ParseResult = llvm::RISCVISAInfo::parseArchString(
+      Arch, /*EnableExperimentalExtension=*/true,
+      /*ExperimentalExtensionVersionCheck=*/false);
+  if (!ParseResult) {
+    // Ignore any error here, we assume it will handled in another place.
+    consumeError(ParseResult.takeError());
+    return false;
+  }
+  auto &ISAInfo = *ParseResult;
+
+  auto CurrentExts = ISAInfo->getExtensions();
+
+  addMultilibFlag(ISAInfo->getXLen() == 32, "-m32", NewFlags);
+  addMultilibFlag(ISAInfo->getXLen() == 64, "-m64", NewFlags);
+
+  // Collect all flags except march=*
+  for (StringRef Flag : Flags) {
+    if (Flag.startswith("!march=") || Flag.startswith("-march="))
+      continue;
+
+    NewFlags.push_back(Flag.str());
+  }
+
+  llvm::StringSet<> AllArchExts;
+  // Reconstruct multi-lib list, and break march option into seperated
+  // extension. e.g. march=rv32im -> +i +m
+  for (auto M : RISCVMultilibSet) {
+    bool Skip = false;
+
+    MultilibBuilder NewMultilib =
+        MultilibBuilder(M.gccSuffix(), M.osSuffix(), M.includeSuffix());
+    for (StringRef Flag : M.flags()) {
+      // Add back the all option except -march.
+      if (!Flag.startswith("-march=")) {
+        NewMultilib.flag(Flag);
+        continue;
+      }
+
+      // Break down -march into individual extension.
+      auto MLConfigParseResult = llvm::RISCVISAInfo::parseArchString(
+          Flag.drop_front(7), /*EnableExperimentalExtension=*/true,
+          /*ExperimentalExtensionVersionCheck=*/false);
+      if (!MLConfigParseResult) {
+        // Ignore any error here, we assume it will handled in another place.
+        llvm::consumeError(MLConfigParseResult.takeError());
+
+        // We might got parsing error if rv32e in the list, we could just skip
+        // that and process the rest of multi-lib configs.
+        Skip = true;
+        continue;
+      }
+      auto &MLConfigISAInfo = *MLConfigParseResult;
+
+      auto MLConfigArchExts = MLConfigISAInfo->getExtensions();
+      for (auto MLConfigArchExt : MLConfigArchExts) {
+        auto ExtName = MLConfigArchExt.first;
+        NewMultilib.flag(Twine("-", ExtName).str());
+
+        if (!AllArchExts.contains(ExtName)) {
+          AllArchExts.insert(ExtName);
+          addMultilibFlag(ISAInfo->hasExtension(ExtName),
+                          Twine("-", ExtName).str(), NewFlags);
+        }
+      }
+
+      // Check the XLEN explicitly.
+      if (MLConfigISAInfo->getXLen() == 32) {
+        NewMultilib.flag("-m32");
+        NewMultilib.flag("!m64");
+      } else {
+        NewMultilib.flag("!m32");
+        NewMultilib.flag("-m64");
+      }
+
+      // Atomic extension must be explicitly checked, soft and hard atomic
+      // operation never co-work correctly.
+      if (!MLConfigISAInfo->hasExtension("a"))
+        NewMultilib.flag("!a");
+    }
+
+    if (Skip)
+      continue;
+
+    NewMultilibs.emplace_back(NewMultilib);
+  }
+
+  // Build an internal used only multi-lib list, used for checking any
+  // compatible multi-lib.
+  MultilibSet NewRISCVMultilibs =
+      MultilibSetBuilder().Either(NewMultilibs).makeMultilibSet();
+
+  if (NewRISCVMultilibs.select(NewFlags, SelectedMultilibs))
+    for (const Multilib &NewSelectedM : SelectedMultilibs)
+      for (auto M : RISCVMultilibSet)
+        // Look up the corresponding multi-lib entry in original multi-lib set.
+        if (M.gccSuffix() == NewSelectedM.gccSuffix())
+          return true;
+
+  return false;
+}
+
 static void findRISCVBareMetalMultilibs(const Driver &D,
                                         const llvm::Triple &TargetTriple,
                                         StringRef Path, const ArgList &Args,
@@ -1727,9 +1851,10 @@ static void findRISCVBareMetalMultilibs(const Driver &D,
   // currently only support the set of multilibs like riscv-gnu-toolchain does.
   // TODO: support MULTILIB_REUSE
   constexpr RiscvMultilib RISCVMultilibSet[] = {
-      {"rv32i", "ilp32"},     {"rv32im", "ilp32"},     {"rv32iac", "ilp32"},
-      {"rv32imac", "ilp32"},  {"rv32imafc", "ilp32f"}, {"rv64imac", "lp64"},
-      {"rv64imafdc", "lp64d"}};
+      {"rv32i", "ilp32"},    {"rv32ic", "ilp32"},     {"rv32if", "ilp32f"},
+      {"rv32im", "ilp32"},   {"rv32iac", "ilp32"},    {"rv32ifc", "ilp32"},
+      {"rv32imac", "ilp32"}, {"rv32imafc", "ilp32f"}, {"rv64imac", "lp64"},
+      {"rv64imac", "lp64"},  {"rv64imafdc", "lp64d"}};
 
   std::vector<MultilibBuilder> Ms;
   for (auto Element : RISCVMultilibSet) {
@@ -1766,7 +1891,8 @@ static void findRISCVBareMetalMultilibs(const Driver &D,
     }
   }
 
-  if (RISCVMultilibs.select(Flags, Result.SelectedMultilibs))
+  if (RISCVMultilibSelect(RISCVMultilibs, MArch, Flags,
+                          Result.SelectedMultilibs))
     Result.Multilibs = RISCVMultilibs;
 }
 
diff --git a/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ic/ilp32/crtbegin.o b/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ic/ilp32/crtbegin.o
new file mode 100644
index 000000000000000..e69de29bb2d1d64
diff --git a/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32if/ilp32f/crtbegin.o b/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32if/ilp32f/crtbegin.o
new file mode 100644
index 000000000000000..e69de29bb2d1d64
diff --git a/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ifc/ilp32/crtbegin.o b/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ifc/ilp32/crtbegin.o
new file mode 100644
index 000000000000000..e69de29bb2d1d64
diff --git a/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ifc/ilp32f/crtbegin.o b/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ifc/ilp32f/crtbegin.o
new file mode 100644
index 000000000000000..e69de29bb2d1d64
diff --git a/clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c b/clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c
new file mode 100644
index 000000000000000..7a1bca76fee8ec4
--- /dev/null
+++ b/clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c
@@ -0,0 +1,86 @@
+// Test case for scanning input of GCC output as multilib config
+// Skip this test on Windows, we can't create a dummy GCC to output
+// multilib config, ExecuteAndWait only execute *.exe file.
+// UNSUPPORTED: system-windows
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32imc -mabi=ilp32 \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMC-ILP32 %s
+// GCC-MULTI-LIB-REUSE-RV32IMC-ILP32: rv32im/ilp32
+// GCC-MULTI-LIB-REUSE-RV32IMC-ILP32-NOT:  {{^.+$}}
+
+// Check rv32imac won't reuse rv32im or rv32ic
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32imac -mabi=ilp32 \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMAC-ILP32 %s
+// GCC-MULTI-LIB-REUSE-RV32IMAC-ILP32: rv32imac/ilp32
+// GCC-MULTI-LIB-REUSE-RV32IMAC-ILP32--NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32ifc -mabi=ilp32f \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IFC-ILP32F %s
+// GCC-MULTI-LIB-REUSE-RV32IFC-ILP32F: rv32if/ilp32f
+// GCC-MULTI-LIB-REUSE-RV32IFC-ILP32F-NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32imafdc -mabi=ilp32f \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMAFDC-ILP32F %s
+// GCC-MULTI-LIB-REUSE-RV32IMAFDC-ILP32F: rv32imafc/ilp32f
+// GCC-MULTI-LIB-REUSE-RV32IMAFDC-ILP32F-NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32imafdc -mabi=ilp32d \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMAFDC-ILP32D %s
+// GCC-MULTI-LIB-REUSE-RV32IMAFDC-ILP32D: .
+// GCC-MULTI-LIB-REUSE-RV32IMAFDC-ILP32D-NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv64imafc -mabi=lp64 \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV64IMAFC-LP64 %s
+// GCC-MULTI-LIB-REUSE-RV64IMAFC-LP64: rv64imac/lp64
+// GCC-MULTI-LIB-REUSE-RV64IMAFC-LP64-NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32imafc_zfh -mabi=ilp32 \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMAFC_ZFH-ILP32 %s
+// GCC-MULTI-LIB-REUSE-RV32IMAFC_ZFH-ILP32: rv32imac/ilp32
+// GCC-MULTI-LIB-REUSE-RV32IMAFC_ZFH-ILP32-NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32i_zvkb -mabi=ilp32 \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32I_ZVKB-ILP32 %s
+// GCC-MULTI-LIB-REUSE-RV32I_ZVKB-ILP32: rv32i/ilp32
+// GCC-MULTI-LIB-REUSE-RV32I_ZVKB-ILP32-NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv64imfc -mabi=lp64 \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV64IMFC-LP64 %s
+// GCC-MULTI-LIB-REUSE-RV64IMFC-LP64: .
+// GCC-MULTI-LIB-REUSE-RV64IMFC-LP64-NOT: {{^.+$}}

llvmbot · 2023-11-29T08:43:49Z

@llvm/pr-subscribers-backend-risc-v

Author: Brandon Wu (4vtomat)

Changes

Extend the multi-lib re-use selection mechanism for RISC-V.
This funciton will try to re-use multi-lib if they are compatible.
Definition of compatible:

ABI must be the same.
multi-lib is a subset of current arch, e.g. multi-lib=march=rv32im
is a subset of march=rv32imc.
march that contains atomic extension can't reuse multi-lib that
doesn't has atomic, vice versa. e.g. multi-lib=march=rv32im and
march=rv32ima are not compatible, because software and hardware
atomic operation can't work together correctly.

Full diff: https://github.com/llvm/llvm-project/pull/73765.diff

6 Files Affected:

(modified) clang/lib/Driver/ToolChains/Gnu.cpp (+130-4)
(added) clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ic/ilp32/crtbegin.o ()
(added) clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32if/ilp32f/crtbegin.o ()
(added) clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ifc/ilp32/crtbegin.o ()
(added) clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ifc/ilp32f/crtbegin.o ()
(added) clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c (+86)

diff --git a/clang/lib/Driver/ToolChains/Gnu.cpp b/clang/lib/Driver/ToolChains/Gnu.cpp
index 16cf5707227264d..ffc53377d97bd02 100644
--- a/clang/lib/Driver/ToolChains/Gnu.cpp
+++ b/clang/lib/Driver/ToolChains/Gnu.cpp
@@ -30,6 +30,7 @@
 #include "llvm/Option/ArgList.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/RISCVISAInfo.h"
 #include "llvm/Support/VirtualFileSystem.h"
 #include "llvm/TargetParser/TargetParser.h"
 #include <system_error>
@@ -1715,6 +1716,129 @@ static void findCSKYMultilibs(const Driver &D, const llvm::Triple &TargetTriple,
     Result.Multilibs = CSKYMultilibs;
 }
 
+/// Extend the multi-lib re-use selection mechanism for RISC-V.
+/// This funciton will try to re-use multi-lib if they are compatible.
+/// Definition of compatible:
+///   - ABI must be the same.
+///   - multi-lib is a subset of current arch, e.g. multi-lib=march=rv32im
+///     is a subset of march=rv32imc.
+///   - march that contains atomic extension can't reuse multi-lib that
+///     doesn't has atomic, vice versa. e.g. multi-lib=march=rv32im and
+///     march=rv32ima are not compatible, because software and hardware
+///     atomic operation can't work together correctly.
+static bool
+RISCVMultilibSelect(const MultilibSet &RISCVMultilibSet, StringRef Arch,
+                    const Multilib::flags_list &Flags,
+                    llvm::SmallVector<Multilib> &SelectedMultilibs) {
+  // Try to find the perfect matching multi-lib first.
+  if (RISCVMultilibSet.select(Flags, SelectedMultilibs))
+    return true;
+
+  llvm::StringMap<bool> FlagSet;
+  Multilib::flags_list NewFlags;
+  std::vector<MultilibBuilder> NewMultilibs;
+
+  auto ParseResult = llvm::RISCVISAInfo::parseArchString(
+      Arch, /*EnableExperimentalExtension=*/true,
+      /*ExperimentalExtensionVersionCheck=*/false);
+  if (!ParseResult) {
+    // Ignore any error here, we assume it will handled in another place.
+    consumeError(ParseResult.takeError());
+    return false;
+  }
+  auto &ISAInfo = *ParseResult;
+
+  auto CurrentExts = ISAInfo->getExtensions();
+
+  addMultilibFlag(ISAInfo->getXLen() == 32, "-m32", NewFlags);
+  addMultilibFlag(ISAInfo->getXLen() == 64, "-m64", NewFlags);
+
+  // Collect all flags except march=*
+  for (StringRef Flag : Flags) {
+    if (Flag.startswith("!march=") || Flag.startswith("-march="))
+      continue;
+
+    NewFlags.push_back(Flag.str());
+  }
+
+  llvm::StringSet<> AllArchExts;
+  // Reconstruct multi-lib list, and break march option into seperated
+  // extension. e.g. march=rv32im -> +i +m
+  for (auto M : RISCVMultilibSet) {
+    bool Skip = false;
+
+    MultilibBuilder NewMultilib =
+        MultilibBuilder(M.gccSuffix(), M.osSuffix(), M.includeSuffix());
+    for (StringRef Flag : M.flags()) {
+      // Add back the all option except -march.
+      if (!Flag.startswith("-march=")) {
+        NewMultilib.flag(Flag);
+        continue;
+      }
+
+      // Break down -march into individual extension.
+      auto MLConfigParseResult = llvm::RISCVISAInfo::parseArchString(
+          Flag.drop_front(7), /*EnableExperimentalExtension=*/true,
+          /*ExperimentalExtensionVersionCheck=*/false);
+      if (!MLConfigParseResult) {
+        // Ignore any error here, we assume it will handled in another place.
+        llvm::consumeError(MLConfigParseResult.takeError());
+
+        // We might got parsing error if rv32e in the list, we could just skip
+        // that and process the rest of multi-lib configs.
+        Skip = true;
+        continue;
+      }
+      auto &MLConfigISAInfo = *MLConfigParseResult;
+
+      auto MLConfigArchExts = MLConfigISAInfo->getExtensions();
+      for (auto MLConfigArchExt : MLConfigArchExts) {
+        auto ExtName = MLConfigArchExt.first;
+        NewMultilib.flag(Twine("-", ExtName).str());
+
+        if (!AllArchExts.contains(ExtName)) {
+          AllArchExts.insert(ExtName);
+          addMultilibFlag(ISAInfo->hasExtension(ExtName),
+                          Twine("-", ExtName).str(), NewFlags);
+        }
+      }
+
+      // Check the XLEN explicitly.
+      if (MLConfigISAInfo->getXLen() == 32) {
+        NewMultilib.flag("-m32");
+        NewMultilib.flag("!m64");
+      } else {
+        NewMultilib.flag("!m32");
+        NewMultilib.flag("-m64");
+      }
+
+      // Atomic extension must be explicitly checked, soft and hard atomic
+      // operation never co-work correctly.
+      if (!MLConfigISAInfo->hasExtension("a"))
+        NewMultilib.flag("!a");
+    }
+
+    if (Skip)
+      continue;
+
+    NewMultilibs.emplace_back(NewMultilib);
+  }
+
+  // Build an internal used only multi-lib list, used for checking any
+  // compatible multi-lib.
+  MultilibSet NewRISCVMultilibs =
+      MultilibSetBuilder().Either(NewMultilibs).makeMultilibSet();
+
+  if (NewRISCVMultilibs.select(NewFlags, SelectedMultilibs))
+    for (const Multilib &NewSelectedM : SelectedMultilibs)
+      for (auto M : RISCVMultilibSet)
+        // Look up the corresponding multi-lib entry in original multi-lib set.
+        if (M.gccSuffix() == NewSelectedM.gccSuffix())
+          return true;
+
+  return false;
+}
+
 static void findRISCVBareMetalMultilibs(const Driver &D,
                                         const llvm::Triple &TargetTriple,
                                         StringRef Path, const ArgList &Args,
@@ -1727,9 +1851,10 @@ static void findRISCVBareMetalMultilibs(const Driver &D,
   // currently only support the set of multilibs like riscv-gnu-toolchain does.
   // TODO: support MULTILIB_REUSE
   constexpr RiscvMultilib RISCVMultilibSet[] = {
-      {"rv32i", "ilp32"},     {"rv32im", "ilp32"},     {"rv32iac", "ilp32"},
-      {"rv32imac", "ilp32"},  {"rv32imafc", "ilp32f"}, {"rv64imac", "lp64"},
-      {"rv64imafdc", "lp64d"}};
+      {"rv32i", "ilp32"},    {"rv32ic", "ilp32"},     {"rv32if", "ilp32f"},
+      {"rv32im", "ilp32"},   {"rv32iac", "ilp32"},    {"rv32ifc", "ilp32"},
+      {"rv32imac", "ilp32"}, {"rv32imafc", "ilp32f"}, {"rv64imac", "lp64"},
+      {"rv64imac", "lp64"},  {"rv64imafdc", "lp64d"}};
 
   std::vector<MultilibBuilder> Ms;
   for (auto Element : RISCVMultilibSet) {
@@ -1766,7 +1891,8 @@ static void findRISCVBareMetalMultilibs(const Driver &D,
     }
   }
 
-  if (RISCVMultilibs.select(Flags, Result.SelectedMultilibs))
+  if (RISCVMultilibSelect(RISCVMultilibs, MArch, Flags,
+                          Result.SelectedMultilibs))
     Result.Multilibs = RISCVMultilibs;
 }
 
diff --git a/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ic/ilp32/crtbegin.o b/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ic/ilp32/crtbegin.o
new file mode 100644
index 000000000000000..e69de29bb2d1d64
diff --git a/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32if/ilp32f/crtbegin.o b/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32if/ilp32f/crtbegin.o
new file mode 100644
index 000000000000000..e69de29bb2d1d64
diff --git a/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ifc/ilp32/crtbegin.o b/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ifc/ilp32/crtbegin.o
new file mode 100644
index 000000000000000..e69de29bb2d1d64
diff --git a/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ifc/ilp32f/crtbegin.o b/clang/test/Driver/Inputs/multilib_riscv_elf_sdk/lib/gcc/riscv64-unknown-elf/8.2.0/rv32ifc/ilp32f/crtbegin.o
new file mode 100644
index 000000000000000..e69de29bb2d1d64
diff --git a/clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c b/clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c
new file mode 100644
index 000000000000000..7a1bca76fee8ec4
--- /dev/null
+++ b/clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c
@@ -0,0 +1,86 @@
+// Test case for scanning input of GCC output as multilib config
+// Skip this test on Windows, we can't create a dummy GCC to output
+// multilib config, ExecuteAndWait only execute *.exe file.
+// UNSUPPORTED: system-windows
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32imc -mabi=ilp32 \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMC-ILP32 %s
+// GCC-MULTI-LIB-REUSE-RV32IMC-ILP32: rv32im/ilp32
+// GCC-MULTI-LIB-REUSE-RV32IMC-ILP32-NOT:  {{^.+$}}
+
+// Check rv32imac won't reuse rv32im or rv32ic
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32imac -mabi=ilp32 \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMAC-ILP32 %s
+// GCC-MULTI-LIB-REUSE-RV32IMAC-ILP32: rv32imac/ilp32
+// GCC-MULTI-LIB-REUSE-RV32IMAC-ILP32--NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32ifc -mabi=ilp32f \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IFC-ILP32F %s
+// GCC-MULTI-LIB-REUSE-RV32IFC-ILP32F: rv32if/ilp32f
+// GCC-MULTI-LIB-REUSE-RV32IFC-ILP32F-NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32imafdc -mabi=ilp32f \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMAFDC-ILP32F %s
+// GCC-MULTI-LIB-REUSE-RV32IMAFDC-ILP32F: rv32imafc/ilp32f
+// GCC-MULTI-LIB-REUSE-RV32IMAFDC-ILP32F-NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32imafdc -mabi=ilp32d \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMAFDC-ILP32D %s
+// GCC-MULTI-LIB-REUSE-RV32IMAFDC-ILP32D: .
+// GCC-MULTI-LIB-REUSE-RV32IMAFDC-ILP32D-NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv64imafc -mabi=lp64 \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV64IMAFC-LP64 %s
+// GCC-MULTI-LIB-REUSE-RV64IMAFC-LP64: rv64imac/lp64
+// GCC-MULTI-LIB-REUSE-RV64IMAFC-LP64-NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32imafc_zfh -mabi=ilp32 \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMAFC_ZFH-ILP32 %s
+// GCC-MULTI-LIB-REUSE-RV32IMAFC_ZFH-ILP32: rv32imac/ilp32
+// GCC-MULTI-LIB-REUSE-RV32IMAFC_ZFH-ILP32-NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv32i_zvkb -mabi=ilp32 \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32I_ZVKB-ILP32 %s
+// GCC-MULTI-LIB-REUSE-RV32I_ZVKB-ILP32: rv32i/ilp32
+// GCC-MULTI-LIB-REUSE-RV32I_ZVKB-ILP32-NOT: {{^.+$}}
+
+// RUN: %clang %s \
+// RUN:   -target riscv64-unknown-elf \
+// RUN:   --gcc-toolchain=%S/Inputs/multilib_riscv_elf_sdk \
+// RUN:   --print-multi-directory \
+// RUN:   -march=rv64imfc -mabi=lp64 \
+// RUN:   | FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV64IMFC-LP64 %s
+// GCC-MULTI-LIB-REUSE-RV64IMFC-LP64: .
+// GCC-MULTI-LIB-REUSE-RV64IMFC-LP64-NOT: {{^.+$}}

clang/lib/Driver/ToolChains/Gnu.cpp

clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c

4vtomat · 2023-12-04T08:35:37Z

Resolved the comments and also fixed the typo of the commit messages.

clang/lib/Driver/ToolChains/Gnu.cpp

Extend the multi-lib re-use selection mechanism for RISC-V. This function will try to re-use multi-lib if they are compatible. Definition of compatible: - ABI must be the same. - multi-lib is a subset of current arch, e.g. multi-lib=march=rv32im is a subset of march=rv32imc. - march that contains atomic extension can't reuse multi-lib that doesn't have atomic, vice versa. e.g. multi-lib=march=rv32im and march=rv32ima are not compatible, because software and hardware atomic operation can't work together correctly.

…oolchain

topperc

LGTM

jayfoad · 2023-12-18T10:23:20Z

The new test is crashing in my Release+Asserts build:

FAIL: Clang :: Driver/riscv-toolchain-gcc-multilib-reuse.c (1081 of 1081)
******************** TEST 'Clang :: Driver/riscv-toolchain-gcc-multilib-reuse.c' FAILED ********************
Exit Code: 2

Command Output (stderr):
--
RUN: at line 1: /home/jayfoad2/llvm-release/bin/clang /home/jayfoad2/git/llvm-project/clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c    -target riscv64-unknown-elf    --gcc-toolchain=/home/jayfoad2/git/llvm-project/clang/test/Driver/Inputs/multilib_riscv_elf_sdk    --print-multi-directory    -march=rv32imc -mabi=ilp32    | /home/jayfoad2/llvm-release/bin/FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMC-ILP32 /home/jayfoad2/git/llvm-project/clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c
+ /home/jayfoad2/llvm-release/bin/FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMC-ILP32 /home/jayfoad2/git/llvm-project/clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c
+ /home/jayfoad2/llvm-release/bin/clang /home/jayfoad2/git/llvm-project/clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c -target riscv64-unknown-elf --gcc-toolchain=/home/jayfoad2/git/llvm-project/clang/test/Driver/Inputs/multilib_riscv_elf_sdk --print-multi-directory -march=rv32imc -mabi=ilp32
clang: /home/jayfoad2/git/llvm-project/clang/lib/Driver/ToolChains/CommonArgs.cpp:2189: void clang::driver::tools::addMultilibFlag(bool, const llvm::StringRef, Multilib::flags_list &): Assertion `Flag.front() == '-'' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /home/jayfoad2/llvm-release/bin/clang /home/jayfoad2/git/llvm-project/clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c -target riscv64-unknown-elf --gcc-toolchain=/home/jayfoad2/git/llvm-project/clang/test/Driver/Inputs/multilib_riscv_elf_sdk --print-multi-directory -march=rv32imc -mabi=ilp32
1.	Compilation construction
 #0 0x00000000070bfaf7 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/jayfoad2/llvm-release/bin/clang+0x70bfaf7)
 #1 0x00000000070bd6ae llvm::sys::RunSignalHandlers() (/home/jayfoad2/llvm-release/bin/clang+0x70bd6ae)
 #2 0x00000000070c01ca SignalHandler(int) Signals.cpp:0:0
 #3 0x00007fc909c42520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #4 0x00007fc909c969fc __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #5 0x00007fc909c969fc __pthread_kill_internal ./nptl/pthread_kill.c:78:10
 #6 0x00007fc909c969fc pthread_kill ./nptl/pthread_kill.c:89:10
 #7 0x00007fc909c42476 gsignal ./signal/../sysdeps/posix/raise.c:27:6
 #8 0x00007fc909c287f3 abort ./stdlib/abort.c:81:7
 #9 0x00007fc909c2871b _nl_load_domain ./intl/loadmsgcat.c:1177:9
#10 0x00007fc909c39e96 (/lib/x86_64-linux-gnu/libc.so.6+0x39e96)
#11 0x0000000007b32257 clang::driver::tools::addMultilibFlag(bool, llvm::StringRef, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>&) (/home/jayfoad2/llvm-release/bin/clang+0x7b32257)
#12 0x0000000007abb016 clang::driver::MultilibBuilder::flag(llvm::StringRef, bool) (/home/jayfoad2/llvm-release/bin/clang+0x7abb016)
#13 0x0000000007b9ddbf findRISCVMultilibs(clang::driver::Driver const&, llvm::Triple const&, llvm::StringRef, llvm::opt::ArgList const&, clang::driver::DetectedMultilibs&) Gnu.cpp:0:0
#14 0x0000000007b95459 clang::driver::toolchains::Generic_GCC::GCCInstallationDetector::ScanGCCForMultilibs(llvm::Triple const&, llvm::opt::ArgList const&, llvm::StringRef, bool) Gnu.cpp:0:0
#15 0x0000000007b9b164 clang::driver::toolchains::Generic_GCC::GCCInstallationDetector::ScanLibDirForGCCTriple(llvm::Triple const&, llvm::opt::ArgList const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, llvm::StringRef, bool, bool, bool) Gnu.cpp:0:0
#16 0x0000000007b9324c clang::driver::toolchains::Generic_GCC::GCCInstallationDetector::init(llvm::Triple const&, llvm::opt::ArgList const&) Gnu.cpp:0:0
#17 0x0000000007bf34ba clang::driver::toolchains::RISCVToolChain::RISCVToolChain(clang::driver::Driver const&, llvm::Triple const&, llvm::opt::ArgList const&) RISCVToolchain.cpp:0:0
#18 0x0000000007a31458 clang::driver::Driver::getToolChain(llvm::opt::ArgList const&, llvm::Triple const&) const (/home/jayfoad2/llvm-release/bin/clang+0x7a31458)
#19 0x0000000007a38bbe clang::driver::Driver::BuildCompilation(llvm::ArrayRef<char const*>) (/home/jayfoad2/llvm-release/bin/clang+0x7a38bbe)
#20 0x0000000004a8a25a clang_main(int, char**, llvm::ToolContext const&) (/home/jayfoad2/llvm-release/bin/clang+0x4a8a25a)
#21 0x0000000004a9bb61 main (/home/jayfoad2/llvm-release/bin/clang+0x4a9bb61)
#22 0x00007fc909c29d90 __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#23 0x00007fc909c29e40 call_init ./csu/../csu/libc-start.c:128:20
#24 0x00007fc909c29e40 __libc_start_main ./csu/../csu/libc-start.c:379:5
#25 0x0000000004a875a5 _start (/home/jayfoad2/llvm-release/bin/clang+0x4a875a5)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /home/jayfoad2/llvm-release/bin/FileCheck -check-prefix=GCC-MULTI-LIB-REUSE-RV32IMC-ILP32 /home/jayfoad2/git/llvm-project/clang/test/Driver/riscv-toolchain-gcc-multilib-reuse.c

--

********************
********************
Failed Tests (1):
  Clang :: Driver/riscv-toolchain-gcc-multilib-reuse.c

antmox · 2023-12-18T10:38:06Z

Hello, looks like it broke some bots too:
clang-aarch64-quick https://lab.llvm.org/buildbot/#/builders/188/builds/39436
clang-armv8-quick https://lab.llvm.org/buildbot/#/builders/245/builds/18162

… toolchain (llvm#73765) Extend the multi-lib re-use selection mechanism for RISC-V. This funciton will try to re-use multi-lib if they are compatible. Definition of compatible: - ABI must be the same. - multi-lib is a subset of current arch, e.g. multi-lib=march=rv32im is a subset of march=rv32imc. - march that contains atomic extension can't reuse multi-lib that doesn't has atomic, vice versa. e.g. multi-lib=march=rv32im and march=rv32ima are not compatible, because software and hardware atomic operation can't work together correctly.

…e-metal toolchain (llvm#73765)

… toolchain (#73765) (#75890) Extend the multi-lib re-use selection mechanism for RISC-V. This funciton will try to re-use multi-lib if they are compatible. Definition of compatible: - ABI must be the same. - multi-lib is a subset of current arch, e.g. multi-lib=march=rv32im is a subset of march=rv32imc. - march that contains atomic extension can't reuse multi-lib that doesn't has atomic, vice versa. e.g. multi-lib=march=rv32im and march=rv32ima are not compatible, because software and hardware atomic operation can't work together correctly.

llvmbot added clang Clang issues not falling into any other category backend:RISC-V clang:driver 'clang' and 'clang++' user-facing binaries. Not 'clang-cl' labels Nov 29, 2023

4vtomat requested review from asb, kito-cheng and topperc November 29, 2023 08:46

4vtomat force-pushed the multilib_reuse branch from 4f6500b to c406872 Compare November 29, 2023 09:10

yetingk reviewed Nov 29, 2023

View reviewed changes

clang/lib/Driver/ToolChains/Gnu.cpp Outdated Show resolved Hide resolved

yetingk reviewed Nov 29, 2023

View reviewed changes

clang/lib/Driver/ToolChains/Gnu.cpp Show resolved Hide resolved

yetingk reviewed Nov 29, 2023

View reviewed changes

clang/lib/Driver/ToolChains/Gnu.cpp Show resolved Hide resolved

yetingk reviewed Nov 29, 2023

View reviewed changes

clang/lib/Driver/ToolChains/Gnu.cpp Outdated Show resolved Hide resolved

topperc reviewed Nov 29, 2023

View reviewed changes

4vtomat force-pushed the multilib_reuse branch from c406872 to fb09422 Compare December 4, 2023 08:35

topperc reviewed Dec 7, 2023

View reviewed changes

clang/lib/Driver/ToolChains/Gnu.cpp Outdated Show resolved Hide resolved

topperc reviewed Dec 7, 2023

View reviewed changes

4vtomat added 2 commits December 12, 2023 20:46

fixup! [RISCV] Implement multi-lib reuse rule for RISC-V bare-metal t…

ecec190

…oolchain

4vtomat force-pushed the multilib_reuse branch from fb09422 to 007fc93 Compare December 13, 2023 05:12

fixup! [RISCV] Implement multi-lib reuse rule for RISC-V bare-metal t…

6caf21f

…oolchain

4vtomat force-pushed the multilib_reuse branch from 007fc93 to 6caf21f Compare December 13, 2023 05:13

topperc approved these changes Dec 13, 2023

View reviewed changes

4vtomat merged commit 111a229 into llvm:main Dec 18, 2023

4vtomat deleted the multilib_reuse branch December 18, 2023 07:52

antmox mentioned this pull request Dec 18, 2023

Revert "[RISCV] Implement multi-lib reuse rule for RISC-V bare-metal … #75789

Merged

4vtomat added a commit to 4vtomat/llvm-project that referenced this pull request Dec 20, 2023

fixup! Recommit [RISCV] Implement multi-lib reuse rule for RISC-V bar…

1f72016

…e-metal toolchain (llvm#73765)

kito-cheng mentioned this pull request Mar 21, 2025

[RISCV] Implement the implications of C extension #132259

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[RISCV] Implement multi-lib reuse rule for RISC-V bare-metal toolchain #73765

[RISCV] Implement multi-lib reuse rule for RISC-V bare-metal toolchain #73765

Uh oh!

4vtomat commented Nov 29, 2023

Uh oh!

llvmbot commented Nov 29, 2023 •

edited

Loading

Uh oh!

llvmbot commented Nov 29, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

4vtomat commented Dec 4, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

topperc left a comment

Uh oh!

jayfoad commented Dec 18, 2023

Uh oh!

antmox commented Dec 18, 2023

Uh oh!

Uh oh!

[RISCV] Implement multi-lib reuse rule for RISC-V bare-metal toolchain #73765

[RISCV] Implement multi-lib reuse rule for RISC-V bare-metal toolchain #73765

Uh oh!

Conversation

4vtomat commented Nov 29, 2023

Uh oh!

llvmbot commented Nov 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented Nov 29, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

4vtomat commented Dec 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

topperc left a comment

Choose a reason for hiding this comment

Uh oh!

jayfoad commented Dec 18, 2023

Uh oh!

antmox commented Dec 18, 2023

Uh oh!

Uh oh!

llvmbot commented Nov 29, 2023 •

edited

Loading

4vtomat commented Dec 4, 2023 •

edited

Loading