This repository was archived by the owner on Sep 10, 2025. It is now read-only.
Define TORCHTEXT_API macro for visibility control #1806
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context:
TorchText uses dual-binding (PyBind11 and TorchBind) to make custom operations available in Python.
The both binding eventually calls the same implementation contained in
libtorchtext.so.The ones bound via PyBind11 (the ones in
torchtext._torchtext) calls intolibtorchtext.so.This means that
libtorchtext.sohas to make the symbols (APIs) used bytorchtext._torchtextvisible.However, the default visibility of symbols in shared libraries are different in Windows.
On Windows all the symbols are by default hidden. To work around this, we use
CMAKE_WINDOWS_EXPORT_ALL_SYMBOLSto expose all the symbols.There is an upper limit of visible symbols one library fine can contain, and this can be problematic in the future.
(Although it is unlikely that torchtext will hit the limit, unless it introduces custom CUDA kernels.)
A better approach is to selectively mark the symbols that should be visible as visible.
Summary of the change set
This commit introduces
TORCHTEXT_APImacro which annotates functions with proper visibility.The core logic was taken from https://github.com/pytorch/pytorch/blob/bcc02769bef1d7b89bec724223284958b7c5b564/c10/macros/Export.h
The behavior is as follow;
For non-Windows: It is always
__attribute__((__visibility__("default")))For Windows:
If the header is included from the compilation unit of
libtorchtext, then it resolves to__declspec(dllexport). otherwise it resolves to__declspec(dllimport).This allows to remove
CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS.