zig cc mingw libraries missing libws2_32.a

[kkartaltepe]

zig cc --target=x86_64-windows-gnu is unable to compile some files which compile correctly with mingw.

Example a.c

#ifdef __linux__
#include <netinet/in.h>
#else
#include <ws2tcpip.h>
#endif
int foo(void) {
  struct in6_addr a = IN6ADDR_ANY_INIT;
  struct in6_addr b = IN6ADDR_LOOPBACK_INIT;
  return IN6_IS_ADDR_UNSPECIFIED(&a) + IN6_IS_ADDR_LOOPBACK(&b);
}

int main(int argc, char **argv) {    
    foo();    
}    

Compiles correctly:

$ gcc -o a.o -Wall example.c
$ x86_64-w64-mingw32-gcc -o a.o -Wall a.c -lws2_32

Zig fails:

$ zig cc --target=x86_64-windows-gnu -o a.o -Wall a.c -lws2_32 
lld-link: error: undefined symbol: IN6_IS_ADDR_UNSPECIFIED
>>> referenced by /home/kk/.cache/zig/o/cb1ef5227d898d0797e8458d2db4fa8e/example.obj:(foo)

lld-link: error: undefined symbol: IN6_IS_ADDR_LOOPBACK
>>> referenced by /home/kk/.cache/zig/o/cb1ef5227d898d0797e8458d2db4fa8e/example.obj:(foo)

[marler8997]

Note that it looks like these are supposed to be defined as "inline" functions.

From my windows 10 SDK in "C:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0\shared\ws2ipdef.h" around line 387:

WS2TCPIP_INLINE
BOOLEAN
IN6_IS_ADDR_UNSPECIFIED(CONST IN6_ADDR *a)
{
    //
    // We can't use the in6addr_any variable, since that would
    // require existing callers to link with a specific library.
    //
    return (BOOLEAN)((a->s6_words[0] == 0) &&
                     (a->s6_words[1] == 0) &&
                     (a->s6_words[2] == 0) &&
                     (a->s6_words[3] == 0) &&
                     (a->s6_words[4] == 0) &&
                     (a->s6_words[5] == 0) &&
                     (a->s6_words[6] == 0) &&
                     (a->s6_words[7] == 0));
}

WS2TCPIP_INLINE
BOOLEAN
IN6_IS_ADDR_LOOPBACK(CONST IN6_ADDR *a)
{
    //
    // We can't use the in6addr_loopback variable, since that would
    // require existing callers to link with a specific library.
    //
    return (BOOLEAN)((a->s6_words[0] == 0) &&
                     (a->s6_words[1] == 0) &&
                     (a->s6_words[2] == 0) &&
                     (a->s6_words[3] == 0) &&
                     (a->s6_words[4] == 0) &&
                     (a->s6_words[5] == 0) &&
                     (a->s6_words[6] == 0) &&
                     (a->s6_words[7] == 0x0100));
}

When I compile this with the MSVC compiler, any reference to these functions is gone. I can run the "Dependencies.exe" tool (https://github.com/lucasg/Dependencies) to see all the imports which prints the output below. Note that it doesn't contain any references to these functions. So you shouldn't be getting any linker errors because these functions may not actualy exists in any Windows "dll" and look like they should be defined in the header files for windows as inline functions or macros.

D:\tools\Dependencies_x64_Release>Dependencies.exe -imports c:\temp\zigbug9364.exe
[-] Import listing for file : c:\temp\zigbug9364.exe
Import from module KERNEL32.dll :
         Function UnhandledExceptionFilter
         Function SetUnhandledExceptionFilter
         Function GetCurrentProcess
         Function TerminateProcess
         Function IsProcessorFeaturePresent
         Function QueryPerformanceCounter
         Function GetCurrentProcessId
         Function GetCurrentThreadId
         Function GetSystemTimeAsFileTime
         Function InitializeSListHead
         Function IsDebuggerPresent
         Function GetStartupInfoW
         Function GetModuleHandleW
         Function WriteConsoleW
         Function RtlUnwind
         Function GetLastError
         Function SetLastError
         Function EnterCriticalSection
         Function LeaveCriticalSection
         Function DeleteCriticalSection
         Function InitializeCriticalSectionAndSpinCount
         Function TlsAlloc
         Function TlsGetValue
         Function TlsSetValue
         Function TlsFree
         Function FreeLibrary
         Function GetProcAddress
         Function LoadLibraryExW
         Function RaiseException
         Function GetStdHandle
         Function WriteFile
         Function GetModuleFileNameW
         Function ExitProcess
         Function GetModuleHandleExW
         Function GetCommandLineA
         Function GetCommandLineW
         Function HeapAlloc
         Function HeapFree
         Function FindClose
         Function FindFirstFileExW
         Function FindNextFileW
         Function IsValidCodePage
         Function GetACP
         Function GetOEMCP
         Function GetCPInfo
         Function MultiByteToWideChar
         Function WideCharToMultiByte
         Function GetEnvironmentStringsW
         Function FreeEnvironmentStringsW
         Function SetEnvironmentVariableW
         Function SetStdHandle
         Function GetFileType
         Function GetStringTypeW
         Function CompareStringW
         Function LCMapStringW
         Function GetProcessHeap
         Function HeapSize
         Function HeapReAlloc
         Function FlushFileBuffers
         Function GetConsoleCP
         Function GetConsoleMode
         Function SetFilePointerEx
         Function CreateFileW
         Function CloseHandle
         Function DecodePointer
[-] Import listing done

[marler8997]

It looks like these functions are in the latest mingw 6.0.0 version, so we just need to update our mingw headers.

[marler8997]

Hmmm, actually it looks like our headers are already up-to-date. One thing I find interesting is that when I preprocess the source file you provide, the definition for one of the missing functions looks like this:

extern inline __attribute__((__gnu_inline__)) int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *a) { return ((a->u.Word[0]==0) && (a->u.Word[1]==0) && (a->u.Word[2]==0) && (a->u.Word[3]==0) && (a->u.Word[4]==0) && (a->u.Word[5]==0) && (a->u.Word[6]==0) && (a->u.Word[7]==0)); }

I believe the error here is the inclusion of extern. The header file source for this functions is:

WS2TCPIP_INLINE int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0)); }

So somehow the problem may be that WS2TCPIP_INLINE is erroneously including extern?

[marler8997]

Found this:

#define WS2TCPIP_INLINE __CRT_INLINE

And:

#ifdef __cplusplus
# define __CRT_INLINE inline
#elif defined(_MSC_VER)
# define __CRT_INLINE __inline
#else
# if ((__MINGW_GNUC_PREREQ(4, 3) || defined(__clang__)) && __STDC_VERSION__ >= 199901L)
#  define __CRT_INLINE extern inline __attribute__((__gnu_inline__))
# else
#  define __CRT_INLINE extern __inline__
# endif
#endif

We're hitting the 3rd case in this defintion define __CRT_INLINE extern inline __attribute__((__gnu_inline__)). I verified that if I define WS2TCPIP_INLINE as inline __attribute__((__gnu_inline__)) (I just removed extern) then the example now compiles and links. So this looks like some sort of bug in the mingw headers.

[marler8997]

Apparently extern inline has a specific meaning, and it's meaning changes depending on which version of libc you are using. So the correct implementation would likely have to detect which version of libc is being used and change the definition based on that, but I'm not sure if we have preprocessor defines that tell us what version of C we are compiling for?

link: https://blahg.josefsipek.net/?tag=inline

NOTE: this could also be a bug in Clang not implementing the correct semantics of extern inline

[marler8997]

Not sure of the fix yet, but one fix might be:

-#define WS2TCPIP_INLINE __CRT_INLINE
+#define WS2TCPIP_INLINE FORCEINLINE

I'm able to compile with this change, but still not sure if it's the right fix.

[marler8997]

This appears to be a bug in Clang's inline implementation. The following program works with gcc but fails with clang:

extern inline __attribute__((__gnu_inline__))
void foo(void) {}

int main(int argc, char *argv[])
{
    foo();
}

GCC works:

$ gcc main.c

Clang (zig cc) fails:

$ zig cc main.c
ld.lld: error: undefined symbol: foo
>>> referenced by main.c
>>>               /home/marler8997/.cache/zig/o/412fc35d19716c55ab1ad7e47350cf7f/main.o:(main)

According to the GNU docs (see gnu_inline from https://gcc.gnu.org/onlinedocs/gcc-6.1.0/gcc/Common-Function-Attributes.html), the function foo in this case will never be emitted.

However, it appears in this case that the combination of clang and gnu_inline creates this position where if clang decides not to inline the function, then we will always get an undefined symbol error during the link.

So my guess here is that when a function is marked as gnu_inline, clang needs to force all calls to be inline.

Or maybe this __attribute__((__gnu_inline__)) is just a bad way to do inlining? Not 100% sure yet.

[marler8997]

Ok I'm doing a 180 here. I think there may have been a miscommunication or misunderstanding with jon_y from IRC. I now think this is a problem with MinGW. The Clang behavior is weird but I believe it is following the spec so it may not actually be a bug. I'm convinced of this now because on MSVC, you can take the address of these functions, but with how MinGW has defined these functions that is impossible.

I'm going to submit the following patch to MinGW's mailing list.

From 2a06367f7c63d5782ae51af162d68ed2e783e389 Mon Sep 17 00:00:00 2001
From: Jonathan Marler <johnnymarler@gmail.com>
Date: Tue, 13 Jul 2021 00:01:03 -0600
Subject: [PATCH] Remove __attribute__((__gnu_inline__)) to allow compiler to
 emit functions

The following example works with MSVC but will fail using the mingw headers:

    #include <ws2tcpip.h>
    int main(int argc, char **argv) {
        return (int)&IN6_IS_ADDR_UNSPECIFIED;
    }

The reason for this is because MinGW is defining this `IN6_IS_ADDR_UNSPECIFIED` function with
`extern inline __attribute__((__gnu_inline__))`. A function defined with these attributes will
NEVER be emitted.  This means you cannot take the address of the function
and if the compiler does not inline a call to it, then you'll get a linker error.

In the Windows SDK in ws2ipdef.h, this function uses the `__inline` attribute

https://docs.microsoft.com/en-us/cpp/cpp/inline-functions-cpp?view=msvc-160

> This keyword tells the compiler that inline expansion is preferred. However, the compiler
> can create a separate instance of the function (instantiate) and create standard calling
> linkages instead of inserting the code inline.  Two cases where this behavior can happen are:
>     * Recursive Functions
>     * Functions that are referred to through a pointer elsewhere in the translation unit.

Note that this function is not defined in any .lib or .dll file in the Windows sdk and/or runtime,
so the only way to be able to take the address of it from the example above is if the compiler
is able to emit the function, or if it's defined elsewhere.

The problem here appears to be the inclusion of `__attribute__((__gnu_inline__))` which is what
tells the compiler NEVER to emit the function definition.  Removing this attribute allows the compiler
to emit the function which allows its address to be taken, and prevents linker errors if the compiler
decides not to inline a call to it.

Note that this applies to multiple functions in the ws2 header files.

Signed-off-by: Jonathan Marler <johnnymarler@gmail.com>
---
 mingw-w64-headers/crt/_mingw.h.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mingw-w64-headers/crt/_mingw.h.in b/mingw-w64-headers/crt/_mingw.h.in
index b7fb99f42..002079910 100644
--- a/mingw-w64-headers/crt/_mingw.h.in
+++ b/mingw-w64-headers/crt/_mingw.h.in
@@ -87,7 +87,7 @@ limitations in handling dllimport attribute.  */
 # define __CRT_INLINE __inline
 #else
 # if ((__MINGW_GNUC_PREREQ(4, 3) || defined(__clang__)) && __STDC_VERSION__ >= 199901L)
-#  define __CRT_INLINE extern inline __attribute__((__gnu_inline__))
+#  define __CRT_INLINE extern inline
 # else
 #  define __CRT_INLINE extern __inline__
 # endif
-- 
2.25.4

[marler8997]

Martin Storsjö from mingw has said on the mailing list that the missing symbols are actually defined in the libws2_32.a file,

https://sourceforge.net/p/mingw-w64/mailman/mingw-w64-public/thread/dc280f-338a-702a-3bc2-a5cce6ebe046%40martin.st/#msg37324166

However, I don't think Zig is including the .a files from mingw. It appears the *.a files are built out of the mingw-w64-crt subdirectory. In there, I can see the definitions for these missing symbols inside C source files which are compiled and combined together into statically linked *.a files, including libws2_32.a. I believe the libws2_32.lib file is a subset of what's included in the libws2_32.a file. The lib file includes imports to functions for the .dll, but the .a file contains both these imports, along with object files that can be statically linked. But when I enable verbose output on zig I don't see it linking to any .a files?

My guess is that we need to also be copying the mingw-w64-crt subdirectory with all its source, and then creating .a files instead of just .lib files and linking to those instead.

[roig]

I isolated an undefined symbol error we have with this code:

#include <ws2tcpip.h>
#include <winsock2.h>
#include "stdint.h"
#include "stdio.h"

int main() {
    struct sockaddr_in6 socket_endpoint;
    memset(&socket_endpoint, 0, sizeof(struct sockaddr_in6));
    socket_endpoint.sin6_family = AF_INET6;
    socket_endpoint.sin6_addr = in6addr_any;
    socket_endpoint.sin6_port = htons(29999);
    printf("%d", socket_endpoint.sin6_addr);
    return 0;
}

• Using visual studio (linking with ws2_32.lib) it compiles and links correctly.
• Using :
  zig cc .\src\ipv6_testing.c -lws2_32
  Results in:

lld-link: error: undefined symbol: in6addr_any
> referenced by C:\Users\Dani\AppData\Local\zig\o\db35fc5242cca1369f3adf2c3efdaaa0\ipv6_testing.obj:(.refptr.in6addr_any)

Is this related to this issue too? If it is, this is a showstopper :(