[SYCL][NVPTX][AMDGCN] Move devicelib cmath to header (#18706)

### Overview

Currently to support C++ builtins in SYCL kernels, we rely on
`libdevice` which
provides implementations for standard library builtins. This library is
built
either to bitcode or SPIR-V and linked in our kernels.

On some targets this causes issues because clang sometimes turns
standard
library calls into LLVM intrinsics that not all targets support.
Specifically on
NVPTX and AMDGCN we can't easily support these intrinsics because we
currently
use implementations provided by CUDA and HIP in the form of a bitcode
library,
which is not something we can use from the LLVM backend.

In upstream LLVM for CUDA and HIP kernels, the way this is handled is
that they
have clang headers providing device-side overloads of C++ library
functions that
hook into the target specific versions of the builtins (for example
`std::sin`
to `__nv_sin`). This way on the device side C++ builtins are hijacked
before
clang can turn them to intrinsics which solves the issue mentioned
above.

This patch is adding the infrastructure to support handling C++ builtins
in SYCL
in the same way as it is done for CUDA and HIP in upstream LLVM. And is
using it
to support `cmath` in NVPTX and AMDGCN compilation.

## Breakdown

* Add `sycl_device_only` attribute: This new attribute allows functions
marked
with it to be treated as device-side overload of existing functions.
This is
what allows us to overload C++ library functions for device in SYCL.
* Remove clang hack to prevent generating LLVM intrinsics from standard
library
builtins for NVPTX and AMDGCN. In theory since this is only moving
`cmath`, the
hack could still be needed, but it looks fine in testing and if we run
into
issues we should just move the problematic builtins to this solution.
The test
`sycl-libdevice-cmath.cpp` was testing this hack, so it was removed.
* `cmath` support for NVPTX and AMDGCN in `libdevice` was disabled. To
limit the
scope of the patch `libdevice` is still fully wired up for these
targets, but it
just won't provide the `cmath` functions.
* Added a `cmath-fallback.h` header providing the device-side math
function
overloads. They are defined using SPIR-V builtins, so in theory this
header
could be used as-is for other targets.
* Use our existing `cmath` stl wrapper to include `cmath-fallback.h` for
NVPTX
and AMDGCN. In upstream LLVM `clang-cuda` always includes with
`-include` the
header with these overloads, using the stl wrappers is a bit more
selective.
* Add `rint` to device lib tests and stl wrapper, this was added in
#18857 but wasn't in E2E testing.


## Compile-time performance

A quick check of compile-time shows that this seems to provide a small
performance improvement. Using two samples, one using cmath (the E2E
`cmath_test.cpp`), and a sample not using cmath, over 10 iterations, I'm
getting the following results:

| Run | Mean | Stdev |
|:--:|:--:|:--:|
|With patch, cmath sample | 4.2229s | 0.0294s |
|With patch, no cmath sample | 5.7484s | 0.0525s |
|Without patch, cmath sample | 4.3817s | 0.0424s |
|Without patch, no cmath sample | 5.7941s | 0.0452s |

Which suggest that the no cmath compile time performance is pretty much
equivalent, and the cmath compile-time performance is faster by roughly
~0.12s.

And this is with the whole `libdevice` setup still in place, so it's
possible this approach could be even more beneficial with more work.

## Future work

* Investigate commented out standard math builtins in
`cmath-fallback.h`, these
weren't defined in libdevice, we should either remove the commented out
lines or
implement them properly.
* Untangle `cmath` and `math.h`, the current `cmath-fallback.h`
implements both
which seems to work fine, but ideally we should split it up.
* Deal with `nearbyint`, this was only implemented for NVPTX and AMDGCN
in
`libdevice`, this patch keeps it the same, but we should look into
proper
support and testing for this.
* Move more of `libdevice` into headers (complex, assert, crt, etc ...).
* Try this approach for SPIR-V or other targets.

---------

Co-authored-by: Alexey Bader <[email protected]>

Author: npmiller

clang/include/clang/Basic/Attr.td

@@ -1607,6 +1607,14 @@ def SYCLDevice : InheritableAttr {
   let Documentation = [SYCLDeviceDocs];
 }
 
+def SYCLDeviceOnly : InheritableAttr {
+  let Spellings = [Clang<"sycl_device_only">];
+  let Subjects = SubjectList<[Function]>;
+  let LangOpts = [SYCLIsDevice, SilentlyIgnoreSYCLIsHost];
+  let Documentation = [SYCLDeviceOnlyDocs];
+}
+def : MutualExclusions<[SYCLDevice, SYCLDeviceOnly]>;
+
 def SYCLGlobalVar : InheritableAttr {
   let Spellings = [GNU<"sycl_global_var">];
   let Subjects = SubjectList<[GlobalStorageNonLocalVar], ErrorDiag>;

clang/include/clang/Basic/AttrDocs.td

@@ -4518,6 +4518,20 @@ implicitly inherit this attribute.
   }];
 }
 
+def SYCLDeviceOnlyDocs : Documentation {
+  let Category = DocCatFunction;
+  let Heading = "sycl_device_only";
+  let Content = [{
+This attribute can only be applied to functions and indicates that the function
+is only available for the device. It allows functions marked with it to
+overload existing functions without the attribute, in which case the overload
+with the attribute will be used on the device side and the overload without
+will be used on the host side. Note: as opposed to ``sycl_device`` this does
+not mark the function as being exported, both attributes are incompatible and
+can't be used together.
+  }];
+}
+
 def RISCVInterruptDocs : Documentation {
   let Category = DocCatFunction;
   let Heading = "interrupt (RISC-V)";

clang/lib/AST/Decl.cpp

@@ -3729,6 +3729,13 @@ unsigned FunctionDecl::getBuiltinID(bool ConsiderWrapperFunctions) const {
       !(BuiltinID == Builtin::BIprintf || BuiltinID == Builtin::BImalloc))
     return 0;
 
+  // SYCL doesn't have a device-side standard library. SYCLDeviceOnlyAttr may
+  // be used to provide device-side definitions of standard functions, so
+  // anything with that attribute shouldn't be treated as a builtin.
+  if (Context.getLangOpts().isSYCL() && hasAttr<SYCLDeviceOnlyAttr>()) {
+    return 0;
+  }
+
   // As AMDGCN implementation of OpenMP does not have a device-side standard
   // library, none of the predefined library functions except printf and malloc
   // should be treated as a builtin i.e. 0 should be returned for them.

clang/lib/CodeGen/CGBuiltin.cpp

@@ -2782,10 +2782,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
       GenerateIntrinsics =
           ConstWithoutErrnoOrExceptions && ErrnoOverridenToFalseWithOpt;
   }
-  bool IsSYCLDeviceWithoutIntrinsics =
-      getLangOpts().SYCLIsDevice &&
-      (getTarget().getTriple().isNVPTX() || getTarget().getTriple().isAMDGCN());
-  if (GenerateIntrinsics && !IsSYCLDeviceWithoutIntrinsics) {
+  if (GenerateIntrinsics) {
     switch (BuiltinIDIfNoAsmLabel) {
     case Builtin::BIacos:
     case Builtin::BIacosf:
@@ -3885,7 +3882,7 @@ RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID,
   case Builtin::BI__builtin_modf:
   case Builtin::BI__builtin_modff:
   case Builtin::BI__builtin_modfl:
-    if (Builder.getIsFPConstrained() || IsSYCLDeviceWithoutIntrinsics)
+    if (Builder.getIsFPConstrained())
       break; // TODO: Emit constrained modf intrinsic once one exists.
     return RValue::get(emitModfBuiltin(*this, E, Intrinsic::modf));
   case Builtin::BI__builtin_isgreater:

clang/lib/CodeGen/CodeGenModule.cpp

@@ -4357,6 +4357,12 @@ void CodeGenModule::EmitGlobal(GlobalDecl GD) {
     }
   }
 
+  // Don't emit 'sycl_device_only' function in SYCL host compilation.
+  if (LangOpts.SYCLIsHost && isa<FunctionDecl>(Global) &&
+      Global->hasAttr<SYCLDeviceOnlyAttr>()) {
+    return;
+  }
+
   if (LangOpts.OpenMP) {
     // If this is OpenMP, check if it is legal to emit this global normally.
     if (OpenMPRuntime && OpenMPRuntime->emitTargetGlobal(GD))
@@ -4446,6 +4452,34 @@ void CodeGenModule::EmitGlobal(GlobalDecl GD) {
     }
   }
 
+  // When using SYCLDeviceOnlyAttr, there can be two functions with the same
+  // mangling, the host function and the device overload. So when compiling for
+  // device we need to make sure we're selecting the SYCLDeviceOnlyAttr
+  // overload and dropping the host overload.
+  if (LangOpts.SYCLIsDevice) {
+    StringRef MangledName = getMangledName(GD);
+    auto DDI = DeferredDecls.find(MangledName);
+    // If we have an existing declaration with the same mangling for this
+    // symbol it may be a SYCLDeviceOnlyAttr case.
+    if (DDI != DeferredDecls.end()) {
+      auto *PreviousGlobal = cast<ValueDecl>(DDI->second.getDecl());
+      // If the host declaration was already processed, replace it with the
+      // device only declaration.
+      if (!PreviousGlobal->hasAttr<SYCLDeviceOnlyAttr>() &&
+          Global->hasAttr<SYCLDeviceOnlyAttr>()) {
+        DeferredDecls[MangledName] = GD;
+        return;
+      }
+
+      // If the device only declaration was already processed, skip the
+      // host declaration.
+      if (PreviousGlobal->hasAttr<SYCLDeviceOnlyAttr>() &&
+          !Global->hasAttr<SYCLDeviceOnlyAttr>()) {
+        return;
+      }
+    }
+  }
+
   // clang::ParseAST ensures that we emit the SYCL devices at the end, so
   // anything that is a device (or indirectly called) will be handled later.
   if (LangOpts.SYCLIsDevice && MustBeEmitted(Global)) {

clang/lib/Sema/SemaDecl.cpp

@@ -1486,6 +1486,17 @@ void Sema::ActOnExitFunctionContext() {
 static bool AllowOverloadingOfFunction(const LookupResult &Previous,
                                        ASTContext &Context,
                                        const FunctionDecl *New) {
+  // SYCLDeviceOnlyAttr allows device side overloads of SYCL function, but it
+  // is incompatible with SYCLDeviceAttr, so don't allow overloads when both
+  // attributes are present.
+  if (Context.getLangOpts().isSYCL() &&
+      Previous.getResultKind() == LookupResultKind::Found &&
+      ((New->hasAttr<SYCLDeviceOnlyAttr>() &&
+        Previous.getFoundDecl()->hasAttr<SYCLDeviceAttr>()) ||
+       (New->hasAttr<SYCLDeviceAttr>() &&
+        Previous.getFoundDecl()->hasAttr<SYCLDeviceOnlyAttr>())))
+    return false;
+
   if (Context.getLangOpts().CPlusPlus || New->hasAttr<OverloadableAttr>())
     return true;
 
@@ -3702,6 +3713,11 @@ bool Sema::MergeFunctionDecl(FunctionDecl *New, NamedDecl *&OldD, Scope *S,
     return true;
   }
 
+  // Never merge SYCLDeviceOnlyAttr functions in their host variant
+  if (getLangOpts().isSYCL() &&
+      Old->hasAttr<SYCLDeviceOnlyAttr>() != New->hasAttr<SYCLDeviceOnlyAttr>())
+    return false;
+
   diag::kind PrevDiag;
   SourceLocation OldLocation;
   std::tie(PrevDiag, OldLocation) =
@@ -7354,6 +7370,10 @@ static bool isIncompleteDeclExternC(Sema &S, const T *D) {
     if (S.getLangOpts().CUDA && (D->template hasAttr<CUDADeviceAttr>() ||
                                  D->template hasAttr<CUDAHostAttr>()))
       return false;
+
+    // So does SYCL's device_only attribute.
+    if (S.getLangOpts().isSYCL() && D->template hasAttr<SYCLDeviceOnlyAttr>())
+      return false;
   }
   return D->isExternC();
 }

clang/lib/Sema/SemaDeclAttr.cpp

@@ -7224,6 +7224,9 @@ ProcessDeclAttribute(Sema &S, Scope *scope, Decl *D, const ParsedAttr &AL,
   case ParsedAttr::AT_SYCLDevice:
     S.SYCL().handleSYCLDeviceAttr(D, AL);
     break;
+  case ParsedAttr::AT_SYCLDeviceOnly:
+    handleSimpleAttribute<SYCLDeviceOnlyAttr>(S, D, AL);
+    break;
   case ParsedAttr::AT_SYCLScope:
     S.SYCL().handleSYCLScopeAttr(D, AL);
     break;

clang/lib/Sema/SemaOverload.cpp

@@ -1629,6 +1629,23 @@ static bool IsOverloadOrOverrideImpl(Sema &SemaRef, FunctionDecl *New,
     }
   }
 
+  // Allow overloads with SYCLDeviceOnlyAttr
+  if (SemaRef.getLangOpts().isSYCL() && (Old->hasAttr<SYCLDeviceOnlyAttr>() !=
+                                         New->hasAttr<SYCLDeviceOnlyAttr>())) {
+    // SYCLDeviceOnlyAttr and SYCLDeviceAttr functions can't overload
+    if (((New->hasAttr<SYCLDeviceOnlyAttr>() &&
+          Old->hasAttr<SYCLDeviceAttr>()) ||
+         (New->hasAttr<SYCLDeviceAttr>() &&
+          Old->hasAttr<SYCLDeviceOnlyAttr>()))) {
+      SemaRef.Diag(New->getLocation(), diag::err_redefinition)
+          << New->getDeclName();
+      SemaRef.notePreviousDefinition(Old, New->getLocation());
+      return false;
+    }
+
+    return true;
+  }
+
   // The signatures match; this is not an overload.
   return false;
 }
@@ -11020,6 +11037,15 @@ bool clang::isBetterOverloadCandidate(
            S.CUDA().IdentifyPreference(Caller, Cand2.Function);
   }
 
+  // In SYCL device compilation mode prefer the overload with the
+  // SYCLDeviceOnly attribute.
+  if (S.getLangOpts().SYCLIsDevice && Cand1.Function && Cand2.Function) {
+    if (Cand1.Function->hasAttr<SYCLDeviceOnlyAttr>() !=
+        Cand2.Function->hasAttr<SYCLDeviceOnlyAttr>()) {
+      return Cand1.Function->hasAttr<SYCLDeviceOnlyAttr>();
+    }
+  }
+
   // General member function overloading is handled above, so this only handles
   // constructors with address spaces.
   // This only handles address spaces since C++ has no other
@@ -11374,6 +11400,15 @@ OverloadingResult OverloadCandidateSet::BestViableFunctionImpl(
   if (S.getLangOpts().CUDA)
     CudaExcludeWrongSideCandidates(S, Candidates);
 
+  // In SYCL host compilation remove candidates marked SYCLDeviceOnly.
+  if (S.getLangOpts().SYCLIsHost) {
+    auto IsDeviceCand = [&](const OverloadCandidate *Cand) {
+      return Cand->Viable && Cand->Function &&
+             Cand->Function->hasAttr<SYCLDeviceOnlyAttr>();
+    };
+    llvm::erase_if(Candidates, IsDeviceCand);
+  }
+
   Best = end();
   for (auto *Cand : Candidates) {
     Cand->Best = false;

clang/test/CodeGenSYCL/sycl-device-only.cpp

@@ -0,0 +1,37 @@
+// RUN: %clang_cc1 -fsycl-is-device -triple spir64-unknown-unknown -disable-llvm-passes -emit-llvm %s -o - | FileCheck %s --check-prefix=CHECKD
+// RUN: %clang_cc1 -fsycl-is-host -triple spir64-unknown-unknown -disable-llvm-passes -emit-llvm %s -o - | FileCheck %s --check-prefix=CHECKH
+// Test code generation for sycl_device_only attribute.
+
+// Verify that the device overload is used on device.
+//
+// CHECK-LABEL: _Z3fooi
+// CHECKH: %add = add nsw i32 %0, 10
+// CHECKD: %add = add nsw i32 %0, 20
+int foo(int a) { return a + 10; }
+__attribute__((sycl_device_only)) int foo(int a) { return a + 20; }
+
+// Use a `sycl_device` function as entry point
+__attribute__((sycl_device)) int bar(int b) { return foo(b); }
+
+// Verify that the order of declaration doesn't change the behavior.
+//
+// CHECK-LABEL: _Z3fooswapi
+// CHECKH: %add = add nsw i32 %0, 10
+// CHECKD: %add = add nsw i32 %0, 20
+__attribute__((sycl_device_only)) int fooswap(int a) { return a + 20; }
+int fooswap(int a) { return a + 10; }
+
+// Use a `sycl_device` function as entry point.
+__attribute__((sycl_device)) int barswap(int b) { return fooswap(b); }
+
+// Verify that in extern C the attribute enables mangling.
+extern "C" {
+// CHECK-LABEL: _Z3fooci
+// CHECKH: %add = add nsw i32 %0, 10
+// CHECKD: %add = add nsw i32 %0, 20
+int fooc(int a) { return a + 10; }
+__attribute__((sycl_device_only)) int fooc(int a) { return a + 20; }
+
+// Use a `sycl_device` function as entry point.
+__attribute__((sycl_device)) int barc(int b) { return fooc(b); }
+}