[SR-359] Undef. symbols when linking to Foundation (ARM)

[hpux735]

Previous ID	SR-359
Radar	None
Original Reporter	@hpux735
Type	Sub-task
Status	Resolved
Resolution	Done

Additional Detail from JIRA

Votes	1
Component/s	Foundation, Standard Library
Labels	Sub-task, Linux, arm, armhf
Assignee	@hpux735
Priority	Medium

md5: d10c8c3be55a3b67735aede67f24479b

Parent-Task:

• SR-40 Port Swift to Linux on Raspberry Pi

Issue Description:

Applications linking to Framework on Linux-ARM fail to link. There are (at least) three undefined symbols:

{{wdillon@tegra-ubuntu:~$ swiftc frameworkTest.swift
/opt/apple/usr/lib/swift/linux/libFoundation.so: undefined reference to `TTSf4s_s_s_nTFs18_fatalErrorMessageFTVs12StaticStringS_S_Su_T*'
/opt/apple/usr/lib/swift/linux/libFoundation.so: undefined reference to `_TFVs13_StringBufferg9usedCountSi'
/opt/apple/usr/lib/swift/linux/libFoundation.so: undefined reference to `TTSf4sTFVs11_StringCore15_invariantCheckfT_T*'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)
wdillon@tegra-ubuntu:~$
}}

The demangled references all seem to call back into string:

function signature specialization <Arg[0] = Exploded, Arg[1] = Exploded, Arg[2] = Exploded> of Swift._fatalErrorMessage (Swift.StaticString, Swift.StaticString, Swift.StaticString, Swift.UInt) -> ()
Swift._StringBuffer.usedCount.getter : Swift.Int
function signature specialization <Arg[0] = Exploded> of Swift._StringCore._invariantCheck () -> ()

Swift._fatalErrorMessage() exists in stdlib/public/core/AssertCommon.swift
The comment in the implementation of _fatalErrorMessage says that it should not be used outside of the implementation of stdlib. It's not clear why foundation would be referencing it.

[phausler]

Is this linking the swift standard library to the product? a verbose compile might be helpful to run here to verify.

[hpux735]

There's an interesting thing happening. I'm using Joe Bell's debian package on one machine, and the products I've built on another, and they behave differently. I noticed that Joe's version version builds and links executables against foundation, but the dredded reloc type 0x03 appears. So, I copied my version (linked with -Wl,-Bsymbolic) of libFoundation.so in place of his (leaving everything else identical) and the executable fails to build with the missing symbols:

{{
wdillon@tegra-ubuntu:~$ sudo mv /usr/lib/swift/linux/libFoundation.so-normal /usr/lib/swift/linux/libFoundation.so
wdillon@tegra-ubuntu:~$ swiftc -v frameworkTest.swift
Swift version 2.2-dev (LLVM 3ebdbb2c7e, Clang f66c5bb67b, Swift 137d201de4)
Target: armv7-unknown-linux-gnueabihf
/usr/bin/swift -frontend -c -primary-file frameworkTest.swift -target armv7-unknown-linux-gnueabihf -disable-objc-interop -color-diagnostics -module-name frameworkTest -o /tmp/frameworkTest-6fee2e.o
/usr/bin/swift-autolink-extract /tmp/frameworkTest-6fee2e.o -o /tmp/frameworkTest-e6b0f4.autolink
/usr/bin/clang++ /tmp/frameworkTest-6fee2e.o -L /usr/lib/swift/linux --target=armv7-unknown-linux-gnueabihf -Xlinker -rpath -Xlinker /usr/lib/swift/linux -lswiftCore @/tmp/frameworkTest-e6b0f4.autolink -Xlinker -T /usr/lib/swift/linux/armv7/swift.ld -o frameworkTest
wdillon@tegra-ubuntu:~$ ./frameworkTest
./frameworkTest: error while loading shared libraries: /usr/lib/swift/linux/libFoundation.so: unexpected reloc type 0x03
wdillon@tegra-ubuntu:~$ sudo mv /usr/lib/swift/linux/libFoundation.so-symbolic /usr/lib/swift/linux/libFoundation.so
wdillon@tegra-ubuntu:~$ swiftc -v frameworkTest.swift
Swift version 2.2-dev (LLVM 3ebdbb2c7e, Clang f66c5bb67b, Swift 137d201de4)
Target: armv7-unknown-linux-gnueabihf
/usr/bin/swift -frontend -c -primary-file frameworkTest.swift -target armv7-unknown-linux-gnueabihf -disable-objc-interop -color-diagnostics -module-name frameworkTest -o /tmp/frameworkTest-7a1265.o
/usr/bin/swift-autolink-extract /tmp/frameworkTest-7a1265.o -o /tmp/frameworkTest-1d91dc.autolink
/usr/bin/clang++ /tmp/frameworkTest-7a1265.o -L /usr/lib/swift/linux --target=armv7-unknown-linux-gnueabihf -Xlinker -rpath -Xlinker /usr/lib/swift/linux -lswiftCore @/tmp/frameworkTest-1d91dc.autolink -Xlinker -T /usr/lib/swift/linux/armv7/swift.ld -o frameworkTest
/usr/lib/swift/linux/libFoundation.so: undefined reference to `TFs18_fatalErrorMessageFTVs12StaticStringS_S_Su_T'
/usr/lib/swift/linux/libFoundation.so: undefined reference to `TFs16_assertionFailedFTVs12StaticStringSSS_Su_T'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)
wdillon@tegra-ubuntu:~$
}}

Another interesting note is that it appears that the symbol is available in libswiftCore.so:

{{
wdillon@tegra-ubuntu:_{$ objdump -x /usr/lib/swift/linux/libFoundation.so |grep TFs18_fatalErrorMessageFTVs12StaticStringS_S_Su_T
00000000 F UND 00000000 TFs18_fatalErrorMessageFTVs12StaticStringS_S_Su_T
wdillon@tegra-ubuntu:}$ objdump -x /usr/lib/swift/linux/ |grep TFs18_fatalErrorMessageFTVs12StaticStringS_S_Su_T
armv7/ libFoundation.so libswiftCore.so libswiftGlibc.so libXCTest.so
wdillon@tegra-ubuntu:~$ objdump -x /usr/lib/swift/linux/libswiftCore.so |grep TFs18_fatalErrorMessageFTVs12StaticStringS_S_Su_T
002728b8 g F .text 00000430 TTSf4s_s_s_n_TFs18_fatalErrorMessageFTVs12StaticStringS_S_Su_T
}}

[phausler]

so is it that libFoundation.so needs -Wl,-Bsymbolic? or the stdlib? or both?

[phausler]

So this means that a symbol is being shared across the DSO when it is not being linked with -Bsymbolic and then the scope of that symbol is somehow being dropped from the compilation unit when that is turned on (likely due to inlining).

This may be an artifact of build order or the way Foundation is constructed via the partial compilation instead of the all-in-one-somewhat-like-javac-build method that the SwiftExecutable build pattern is done.

[hpux735]

stdlib is already compiled with -Wl,-Bsymbolic in my fork. Tienex has been tracking down the last few test failures in his fork. swiftlang/swift#608 (comment)

It's not clear whether foundation should be build with symbolic or not. It doesn't work either way. My guess is that it probably should be symbolic. Also, I wonder if there are paths in my version of the .so that don't match the paths that are in Joe's build products.

[hpux735]

I'll have to take your word for it on the linking process. I'm a total linking noob. 🙂

[phausler]

From the CF standpoint it probably wouldn't make much of a difference since it should already be resilient to any failure cases of the -Bsymbolic option

Here is a small diff that would apply that
https://gist.github.com/phausler/0a2f5578675e42e60d69

It builds just fine on ubuntu 14 x86_64
Are there build instructions anywhere for the ARM work that is being done? I have a Tegra K1 board that I can try some things out on (however I highly doubt it can support the linking phase) so I would rather use a cross-compile if possible.

[hpux735]

Using your patch it behaves just like Joe's version: reloc type 0x03. I'm not as confident about the compile command line during foundation build, though, because it doesn't respect the --verbose-build flag to build-script-impl.

Building on ARM is fairly straight-forward. The only trick is getting the right repos (all the documentation in the readme applies).
My current work horse is a Terga K1, and provided it has some swap space, it'll build and link fine (as long as you use the release [-R] mode).

wdillon@raspberrypi:~ $ cat repos.sh
#!/bin/bash
git clone https://github.com/hpux735/swift.git swift
git clone https://github.com/apple/swift-llvm.git llvm
git clone https://github.com/apple/swift-clang.git clang
git clone https://github.com/hpux735/swift-lldb.git lldb
git clone https://github.com/apple/swift-cmark.git cmark
git clone https://github.com/apple/swift-llbuild.git llbuild
git clone https://github.com/apple/swift-package-manager.git swiftpm
git clone https://github.com/apple/swift-corelibs-xctest.git
git clone https://github.com/hpux735/swift-corelibs-foundation.git

cd swift
./utils/build-script -R --foundation

[phausler]

We recently had a few updates to the linking phase that might explain some more issues here; it would be worth it to perhaps see if this is still happening.

[hpux735]

Hi Philippe,

Thanks for the update. Do you have a pointer to the changes? I'd like to try this out, but I want to make sure that I have the changes locally, and that anything that I've messed with doesn't interfere.

[phausler]

This was the commit that helped fix a few issues from the linker:
41eb1b7

[swift-ci]

Comment by Joe (JIRA)

Will,

Let me know when you are a ready for a build - my build area is based upon this at the moment for building on the X15:

#!/bin/bash
git clone https://github.com/hpux735/swift.git swift
git clone https://github.com/apple/swift-llvm.git llvm
git clone https://github.com/apple/swift-clang.git clang
git clone https://github.com/iachievedit/swift-lldb.git lldb
git clone https://github.com/apple/swift-cmark.git cmark
git clone https://github.com/apple/swift-llbuild.git llbuild
git clone https://github.com/apple/swift-package-manager.git swiftpm
git clone https://github.com/apple/swift-corelibs-xctest.git
git clone https://github.com/apple/swift-corelibs-foundation.git

I think I can switch back to your lldb or the apple if the changes have been merged in?
Joe

[hpux735]

Go for it, Joe. I just checked, and I don't think I have any uncommitted or unmerged changes to foundation, so the Apple one is good.

I'm building right now (have been for about 6-7 hours) so, we'll see. 🙂

[hpux735]

I tried it and I'm reliably getting the reloc type error I had been getting with just the stdlib formerly:

wdillon@tegra-ubuntu:~$ cat frameworkTest.swift
import Foundation

print("Foundation imports")
wdillon@tegra-ubuntu:~$ swiftc frameworkTest.swift
wdillon@tegra-ubuntu:~$ ./frameworkTest
./frameworkTest: error while loading shared libraries: /usr/lib/swift/linux/libFoundation.so: unexpected reloc type 0x03

This might actually be a positive result, because I hope it can be fixed in the same way as last time: "-Wl,-Bsymbolic" I'm trying that now.

[hpux735]

This pull request fixes it. (At least on ARMv7)

#212