Make the `CLIPTokenizer`'s `encoder_json_path` variable optional, and use `dict(zip(vocab, range(len(vocab))))` instead

[ProGamerGov]

🚀 Feature

https://github.com/pytorch/text/blob/main/torchtext/transforms.py#L312

In both CLIP and OpenCLIP, the encoder is simply just the vocab run through dict(zip(vocab, range(len(vocab)))), and it doesn't make a ton of sense to require a encoder.json file for this information. The encoder.json file requirement should be optional as the vocab file itself can be used to create the encoder, making the encoder file redundant.

https://github.com/openai/CLIP/blob/main/clip/simple_tokenizer.py#L74
https://github.com/mlfoundations/open_clip/blob/main/src/clip/tokenizer.py#L78

The current clip_encoder.json test asset is just the Python dict created by dict(zip(vocab, range(len(vocab)))), and thus will it makes a useful test for specifying an encoder, it's redundant: https://github.com/pytorch/text/blob/main/test/asset/clip_encoder.json

[parmeet]

cc: @abhinavarora

[abhinavarora]

Thank you @ProGamerGov for raising this issue. I agree with you that this can get redundant. Currently we take both the encoder json and the vocab file to match the output of HuggingFace's CLIP tokenizer which is a popular CLIP implementation. I am creating a PR to address this issue as well as another issue which deals with hardcoding of vocab length in the CLIP implementation.

I will share a PR that will do the following:

1. Make encoder json optional
2. In case encoder json is provided, infer the vocab length from it. This is to deal with the hardcoding in the CLIP implementation
3. Add an optional num_merges parameter to specify the number of merges when encoder json is not provided. This is again to deal with hardcoded parameters.

Let me know if this makes sense to you?

[ProGamerGov]

@abhinavarora Yeah, that makes sense and sounds good to me!

I am hoping to use TorchText's CLIP tokenizer for pytorch/captum's CLIP model tutorial which utilizes one of OpenAI's released CLIP models.

[abhinavarora]

@abhinavarora Yeah, that makes sense and sounds good to me!

@ProGamerGov Here is PR: #1622

[ProGamerGov]

@abhinavarora Thank you for taking the time to implement the changes!