Skip to content

Conversation

@kaixuanliu
Copy link
Contributor

@kaixuanliu kaixuanliu commented Nov 17, 2025

When I use transformers built from source on windows, I meet several errors:

  1. Existing code use / (forward slash) for splitting paths with rsplit("/", 1), but on Windows, the paths might use \ (backslash). When the path contains backslashes, this causes incorrect path joining
  2. On Windows with symbolic links, checkpoint_files[0] might be a symlink, and when you get its dirname and join it with another file, it will create malformed paths
  3. we need to load shards sequentially for limited gpu memory, and also, on windows, the virtual address for memory mapping is limited, current implementation will casue error OSError: The paging file is too small for this operation to complete
    This PR tries to solve the 3 errors mentioned above.
    @Cyrilvallez pls help review, thx!

@kaixuanliu kaixuanliu marked this pull request as draft November 17, 2025 06:23
@kaixuanliu kaixuanliu marked this pull request as ready for review November 17, 2025 06:37
@Rocketknight1
Copy link
Member

@Cyrilvallez it's your code but I know the core maintainers are busy, let me know if you want me to take this one instead!

@Cyrilvallez
Copy link
Member

Hey! I removed this code this week, so the issue should not happen anymore on main! Let me know if you still encounter it though! In which case feel free to reopen the PR or create a new one!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants