Added Hub and Tokenizers libraries.

[greenrazer]

Added Hub and Tokenizers as separate library products in Package.swift. This change should enable selective compilation of specific components while maintaining a single project structure.

Usage Example

Usage is pretty simple, I created an external project and all that was needed was to edit Package.swift.

Specifically, Add the package dependency:

dependencies: [
        .package(url: "https://github.com/huggingface/swift-transformers", branch: "main")
    ],

Then import specific products as dependencies of the desired target:

targets: [
        .target(
            ...
            dependencies: [
                .product(name: "Hub", package: "swift-transformers")
            ]
        ),

Testing

In the external project I copied and ran all the Tokenizer, and Hub tests from swift-transformers.

• Successfully imported both products
• All tests passed:
  • Hub: 14 tests passed in 9.822s
  • Tokenizers: 96 tests passed in 26.948s

[pcuenca]

According to my tests, all the package dependencies are downloaded and resolved (so other dependencies are recursively pulled as well). However, they are not compiled if they don't participate in the targets used by the client package. (Actually, there's a pre-processing step where stubs and headers are generated, but no actual compilation happens).

I tested building with swift build and Xcode, using a dependent package that uses the Tokenizers library (without Models).

[pcuenca]

Suggested change

	.library(name: "Tokenizers", targets: ["Tokenizers", "Models"]),
	.library(name: "Tokenizers", targets: ["Tokenizers"]),

This would pull most of the library, wouldn't it?

[greenrazer]

Yeah, I was getting an error with tokenizers when I didn't have the Models target, but I think that was just an issue with the test. I implemented the above change and fixed the test, and re-tested it in the external project.

[pcuenca]

Superseded by #168