From d0c01e20017a236c44642f4928156df56d468605 Mon Sep 17 00:00:00 2001 From: Alastair Houghton Date: Tue, 4 Jun 2024 14:52:33 +0100 Subject: [PATCH 1/6] Add a Getting Started document for the Static Linux SDK. First pass at a Getting Started document for the Static Linux SDK. rdar://129217209 --- .../articles/static-linux-getting-started.md | 241 ++++++++++++++++++ 1 file changed, 241 insertions(+) create mode 100644 documentation/articles/static-linux-getting-started.md diff --git a/documentation/articles/static-linux-getting-started.md b/documentation/articles/static-linux-getting-started.md new file mode 100644 index 000000000..eed164938 --- /dev/null +++ b/documentation/articles/static-linux-getting-started.md @@ -0,0 +1,241 @@ +--- +layout: page +date: 2024-06-04 12:00:00 +title: Getting Started with the Static Linux SDK +author: [al45tair] +--- + +It's well known that Swift can be used to build software for Apple +platforms such as macOS or iOS, but Swift is also supported on other +platforms, including Linux and Windows. + +Building for Linux is especially interesting because historically +Linux programs written in Swift needed to ensure that a copy of the +Swift runtime --- and all of its dependencies --- was installed on the +target system. Additionally, a program built for a particular +distribution, or even a particular major version of a particular +distribution, would not necessarily run on any other distribution or +in some cases even on a different major version of the same +distribution. + +The Swift Static Linux SDK solves both of these problems by allowing +you to build your program as a _fully statically linked_ executable, +with no external dependencies at all (not even the C library), which +means that it will run on _any_ Linux distribution as the only thing +it depends on is the Linux system call interface. + +Additionally, the Static Linux SDK can be used from any platform +supported by the Swift compiler and package manager; this means that +you can develop and test your program on macOS before building and +deploying it to a Linux-based server, whether running locally or +somewhere in the cloud. + +### Static vs Dynamic Linking + +_Linking_ is the process of taking different pieces of a computer +program and wiring up any references between those pieces. For +_static_ linking, generally speaking those pieces are _object files_, +or _static libraries_ (which are really just collections of object +files). + +For _dynamic_ linking, the pieces are _executables_ and _dynamic +libraries_ (aka dylibs, shared objects, or DLLs). + +There are two key differences between dynamic and static linking: + +* The time at which linking takes place. Static linking happens when + you build your program; dynamic linking happens at runtime. + +* The fact that a static library (or _archive_) is really a collection + of individual object files, whereas a dynamic library is monolithic. + +The latter is important because traditionally, the static linker will +include every object explicitly listed on its command line, but it +will _only_ include an object from a static library if doing so lets +it resolve an unresolved symbolic reference. If you statically link +against a library that you do not actually use, a traditional static +linker will completely discard that library and not include any code +from it in your final binary. + +In practice things can be more complicated --- the static linker may +actually work on the basis of individual _sections_ or _atoms_ from +your object files, so it may in fact be able to discard individual +functions or pieces of data rather than just whole objects. + +### Pros and Cons of Static Linking + +Pros of static linking: + +* No runtime overhead. + +* Only include code from libraries that is actually needed. + +* No need for separately installed dynamic libraries. + +* No versioning issues at runtime. + +Cons of static linking: + +* Programs cannot share code (higher overall memory usage). + +* No way to update dependencies without rebuilding program. + +* Larger executables (though this can be offset by not having to + install separate dynamic libraries). + +On Linux in particular, it's also possible to use static linking to +completely eliminate dependencies on system libraries supplied by the +distribution, resulting in executables that work on any distribution +and can be installed by simply copying. + +### Installing the SDK + +Before you start, **you will need to [install an Open Source toolchain +from swift.org](https://www.swift.org/install/)**. You cannot use the +toolchain provided with Xcode to build programs using the SDK. If you +are using macOS, you will also need to ensure that you use the Swift +compiler from this toolchain by [following the instructions +here](https://www.swift.org/install/macos/#installation-via-swiftorg-package-installer). + +One that is out of the way, actually installing the Static Linux SDK +is easy; at a prompt, enter + +```shell +swift sdk install +``` + +Swift will download and install the SDK on your system. You can get a +list of installed SDKs with + +```shell +swift sdk list +``` + +and it's also possible to remove them using + +```shell +swift sdk remove +``` + +### Your first statically linked Linux program + +First, create a directory to hold your code: + +```console +$ mkdir hello +$ cd hello +``` + +Next, ask Swift to create a new program package for you: + +```console +$ swift package init --type executable +``` + +You can build and run this locally: + +```console +$ swift build +Building for debugging... +[8/8] Applying hello +Build complete! (15.29s) +$ .build/debug/hello +Hello, world! +``` + +But with the Static Linux SDK installed, you can also build Linux +binaries for x86-64 and ARM64 machines: + +```console +$ swift build --sdk x86_64-swift-linux-musl +Building for debugging... +[8/8] Linking hello +Build complete! (2.04s) +$ .build/x86_64-swift-linux-musl/debug/hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped +``` + +```console +$ swift build --sdk aarch64-swift-linux-musl +Building for debugging... +[8/8] Linking hello +Build complete! (2.00s) +$ file .build/aarch64-swift-linux-musl/debug/hello +.build/aarch64-swift-linux-musl/debug/hello: ELF 64-bit LSB +executable, ARM aarch64, version 1 (SYSV), statically linked, with +debug_info, not stripped +``` + +These can be copied to an appropriate Linux-based system and executed: + +```console +$ scp .build/x86_64-swift-linux-musl/debug/hello linux:~/hello +$ ssh linux ~/hello +Hello, world! +``` + +### What about package dependencies? + +Swift packages that make use of Foundation or Swift NIO should just +work. If you try to use a package that uses the C library, however, +you may have a little work to do. Such packages often contain files +with code like the following: + +```swift +#if os(macOS) || os(iOS) +import Darwin +#elseif os(Linux) +import Glibc +#elseif os(Windows) +import ucrt +#else +#error(Unknown platform) +#endif +``` + +The Static Linux SDK does not use Glibc; instead, it is built on top +of an alternative C library for Linux called +[Musl](https://musl-libc.org). We chose this approach for two +reasons: + +1. Musl has excellent support for static linking. + +2. Musl is permissively licensed, which makes it easy to distribute + executables statically linked with it. + +If you are using such a dependency, you will therefore need to adjust +it to import the `Musl` module instead of the `Glibc` module: + +```swift +#if os(macOS) || os(iOS) +import Darwin +#elseif canImport(Glibc) +import Glibc +#elseif canImport(Musl) +import Musl +#elseif os(Windows) +import ucrt +#else +#error(Unknown platform) +#endif +``` + +Very occasionally there might be a difference between the way a C +library type gets imported between Musl and Glibc; this sometimes +happens if someone has added nullability annotations, or where a +pointer type is using a forward-declared `struct` for which no actual +definition is ever provided. Usually the problem will be obvious --- +a function argument or result will be `Optional` in one case and +non-`Optional` in another, or a pointer type will be imported as +`OpaquePointer` rather than `UnsafePointer`. + +If you do find yourself needing to make these kinds of adjustments, +you can make your local copy of the package dependency editable by +doing + +```console +$ swift package edit SomePackage +``` + +and then editing the files in the `Packages` directory that appears in +your program's source directory. You may wish to consider raising PRs +upstream with any fixes you may have. From e8701eeca9e6bd1850896e06341bb3cd52301f39 Mon Sep 17 00:00:00 2001 From: Alastair Houghton Date: Thu, 6 Jun 2024 15:51:33 +0100 Subject: [PATCH 2/6] Add a link to the Articles section --- _data/documentation.yaml | 5 +++++ documentation/articles/static-linux-getting-started.md | 4 +--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/_data/documentation.yaml b/_data/documentation.yaml index 4e943d374..65dd32f6a 100644 --- a/_data/documentation.yaml +++ b/_data/documentation.yaml @@ -58,6 +58,11 @@ url: /documentation/concurrency/ description: | Prepare for Swift 6 by enabling complete concurrency checking in your SwiftPM packages, Xcode projects, and CI scripts. + - title: Getting Started with the Static Linux SDK + url: /documentation/articles/static-linux-getting-started.html + description: | + Learn how to get started building binaries for Linux with no system dependencies (not even the Swift runtime or C library). + Even better, you can do this from any system with a Swift toolchain, allowing you to develop on macOS or Windows and easily deploy to Linux when you go to production. #---------------------------------------------------- - header: Contributing pages: diff --git a/documentation/articles/static-linux-getting-started.md b/documentation/articles/static-linux-getting-started.md index eed164938..7abae06ae 100644 --- a/documentation/articles/static-linux-getting-started.md +++ b/documentation/articles/static-linux-getting-started.md @@ -160,9 +160,7 @@ Building for debugging... [8/8] Linking hello Build complete! (2.00s) $ file .build/aarch64-swift-linux-musl/debug/hello -.build/aarch64-swift-linux-musl/debug/hello: ELF 64-bit LSB -executable, ARM aarch64, version 1 (SYSV), statically linked, with -debug_info, not stripped +.build/aarch64-swift-linux-musl/debug/hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, with debug_info, not stripped ``` These can be copied to an appropriate Linux-based system and executed: From 285263c1c8e3e36055eb8711b410050524aaf876 Mon Sep 17 00:00:00 2001 From: Alastair Houghton Date: Thu, 6 Jun 2024 17:07:25 +0100 Subject: [PATCH 3/6] Fixed a missing line, and use console everywhere. --- .../articles/static-linux-getting-started.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/documentation/articles/static-linux-getting-started.md b/documentation/articles/static-linux-getting-started.md index 7abae06ae..b164757e6 100644 --- a/documentation/articles/static-linux-getting-started.md +++ b/documentation/articles/static-linux-getting-started.md @@ -100,21 +100,21 @@ here](https://www.swift.org/install/macos/#installation-via-swiftorg-package-ins One that is out of the way, actually installing the Static Linux SDK is easy; at a prompt, enter -```shell -swift sdk install +```console +$ swift sdk install ``` Swift will download and install the SDK on your system. You can get a list of installed SDKs with -```shell -swift sdk list +```console +$ swift sdk list ``` and it's also possible to remove them using -```shell -swift sdk remove +```console +$ swift sdk remove ``` ### Your first statically linked Linux program @@ -151,7 +151,8 @@ $ swift build --sdk x86_64-swift-linux-musl Building for debugging... [8/8] Linking hello Build complete! (2.04s) -$ .build/x86_64-swift-linux-musl/debug/hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped +$ file .build/x86_64-swift-linux-musl/debug/hello +.build/x86_64-swift-linux-musl/debug/hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped ``` ```console From d0b696c6ae6e7944965e86cb67ffb400f18e4192 Mon Sep 17 00:00:00 2001 From: Alastair Houghton Date: Fri, 7 Jun 2024 18:03:04 +0100 Subject: [PATCH 4/6] Added a note about versions --- documentation/articles/static-linux-getting-started.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/documentation/articles/static-linux-getting-started.md b/documentation/articles/static-linux-getting-started.md index b164757e6..5d23f2744 100644 --- a/documentation/articles/static-linux-getting-started.md +++ b/documentation/articles/static-linux-getting-started.md @@ -97,6 +97,11 @@ are using macOS, you will also need to ensure that you use the Swift compiler from this toolchain by [following the instructions here](https://www.swift.org/install/macos/#installation-via-swiftorg-package-installer). +> **Note**: The toolchain must match the version of the Static Linux +> SDK that you install. The Static Linux SDK includes the +> corresponding Swift version in its filename to help identify the +> correct version of the SDK. + One that is out of the way, actually installing the Static Linux SDK is easy; at a prompt, enter From ed5b3415936005fa8fb72305a9d2a1a9e8b38283 Mon Sep 17 00:00:00 2001 From: Alastair Houghton Date: Fri, 7 Jun 2024 19:24:37 +0100 Subject: [PATCH 5/6] Add SDK URL --- .../articles/static-linux-getting-started.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/documentation/articles/static-linux-getting-started.md b/documentation/articles/static-linux-getting-started.md index 5d23f2744..97f81bfd6 100644 --- a/documentation/articles/static-linux-getting-started.md +++ b/documentation/articles/static-linux-getting-started.md @@ -106,9 +106,21 @@ One that is out of the way, actually installing the Static Linux SDK is easy; at a prompt, enter ```console -$ swift sdk install +$ swift sdk install ``` +giving the URL or filename at which the SDK can be found. + +For instance, assuming you have installed the +`swift-6.0-DEVELOPMENT-SNAPSHOT-2024-06-06-a` toolchain, you would +need to enter + +```console +$ swift sdk install https://download.swift.org/development/static-sdk/swift-DEVELOPMENT-SNAPSHOT-2024-06-06-a/swift-DEVELOPMENT-SNAPSHOT-2024-06-06-a_static-linux-0.0.1.artifactbundle.tar.gz +``` + +to install the corresponding Static Linux SDK. + Swift will download and install the SDK on your system. You can get a list of installed SDKs with From 1bd4f5f91b879a15c43f5edd9bb75a5280db32fe Mon Sep 17 00:00:00 2001 From: Alastair Houghton Date: Fri, 7 Jun 2024 21:34:28 +0100 Subject: [PATCH 6/6] Updated after review comments --- .../articles/static-linux-getting-started.md | 50 +++++++++++-------- 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/documentation/articles/static-linux-getting-started.md b/documentation/articles/static-linux-getting-started.md index 97f81bfd6..aec714777 100644 --- a/documentation/articles/static-linux-getting-started.md +++ b/documentation/articles/static-linux-getting-started.md @@ -9,9 +9,9 @@ It's well known that Swift can be used to build software for Apple platforms such as macOS or iOS, but Swift is also supported on other platforms, including Linux and Windows. -Building for Linux is especially interesting because historically +Building for Linux is especially interesting because, historically, Linux programs written in Swift needed to ensure that a copy of the -Swift runtime --- and all of its dependencies --- was installed on the +Swift runtime---and all of its dependencies---was installed on the target system. Additionally, a program built for a particular distribution, or even a particular major version of a particular distribution, would not necessarily run on any other distribution or @@ -57,7 +57,7 @@ against a library that you do not actually use, a traditional static linker will completely discard that library and not include any code from it in your final binary. -In practice things can be more complicated --- the static linker may +In practice, things can be more complicated---the static linker may actually work on the basis of individual _sections_ or _atoms_ from your object files, so it may in fact be able to discard individual functions or pieces of data rather than just whole objects. @@ -90,19 +90,25 @@ and can be installed by simply copying. ### Installing the SDK -Before you start, **you will need to [install an Open Source toolchain -from swift.org](https://www.swift.org/install/)**. You cannot use the -toolchain provided with Xcode to build programs using the SDK. If you -are using macOS, you will also need to ensure that you use the Swift -compiler from this toolchain by [following the instructions -here](https://www.swift.org/install/macos/#installation-via-swiftorg-package-installer). +Before you start, it's important to note: -> **Note**: The toolchain must match the version of the Static Linux -> SDK that you install. The Static Linux SDK includes the -> corresponding Swift version in its filename to help identify the -> correct version of the SDK. +* You will need to [install an Open Source toolchain from + swift.org](https://www.swift.org/install/). -One that is out of the way, actually installing the Static Linux SDK +* You cannot use the toolchain provided with Xcode to build programs + using the SDK. + +* If you are using macOS, you will also need to ensure that you use + the Swift compiler from this toolchain by [following the + instructions + here](https://www.swift.org/install/macos/#installation-via-swiftorg-package-installer). + +* The toolchain must match the version of the Static Linux SDK that + you install. The Static Linux SDK includes the corresponding Swift + version in its filename to help identify the correct version of the + SDK. + +Once that is out of the way, actually installing the Static Linux SDK is easy; at a prompt, enter ```console @@ -235,14 +241,14 @@ import ucrt #endif ``` -Very occasionally there might be a difference between the way a C -library type gets imported between Musl and Glibc; this sometimes -happens if someone has added nullability annotations, or where a -pointer type is using a forward-declared `struct` for which no actual -definition is ever provided. Usually the problem will be obvious --- -a function argument or result will be `Optional` in one case and -non-`Optional` in another, or a pointer type will be imported as -`OpaquePointer` rather than `UnsafePointer`. +Occasionally there might be a difference between the way a C library +type gets imported between Musl and Glibc; this sometimes happens if +someone has added nullability annotations, or where a pointer type is +using a forward-declared `struct` for which no actual definition is +ever provided. Usually the problem will be obvious---a function +argument or result will be `Optional` in one case and non-`Optional` +in another, or a pointer type will be imported as `OpaquePointer` +rather than `UnsafePointer`. If you do find yourself needing to make these kinds of adjustments, you can make your local copy of the package dependency editable by