Remove <unk> token and index from experimental Vocab

[zhangguanheng66]

This PR is to remove the default '<unk>' token along with the index from experimental.vocab. Fix #1016

In the experimental vocabulary, there will be no special symbols or user reserved symbols. Instead, we add a builtin index for the default scenario, and users are required to call set_default_index func explicitly to reset the default index. If not reset, the vocabulary will throw out error message for the default scenario. With the set_default_index function, users will have the flexibility to have or not have default index. For the special symbols (e.g. '<unk>', '<pad>'), users should insert the tokens with the existing method self.insert_token(token: str, index: int). Later on, when users need the index of the special symbols, they can obtain them by calling the vocab instance. For example:

vocab.insert_token('<unk>', 0)
vocab.set_fallback_index(0)
vocab.insert_token('<pad>', 1)
print(vocab('not_in_vocab_token'), vocab('<pad>'))
>>> 0, 1

[zhangguanheng66]

cc @bentrevett Please review the PR and let us know if you have any other suggestions.

[bentrevett]

@zhangguanheng66 All looks good to me.

[cpuhrsch]

Maybe "default" is a better name than "fallback" since it's akin to the default kwarg passed to dict.get.

[cpuhrsch]

You probably also want to check what numbers these tokens correspond to

        self.assertEqual(v['not_in_it'], 0)
        self.assertEqual(v['<unk>'], 0)

[cpuhrsch]

Related to this would be a test that verifies the behavior of insert_token(<unk>) if <unk> is already part of the vocabulary.

[cpuhrsch]

I think it's better if this were to return "None". You should be able to do this easily by using c10::optional<int64_t> instead of int64_t in the C++ code.

[cpuhrsch]

As an aside, while we're introducing this for the Vocab we should probably as a follow-up also introduce the same concepts to the Vectors class

[cpuhrsch]

Before we merge this we also need to support resassignment. A special token might show up in the dataset and ends up inadvertently being mapped to the wrong index. For example, might show up in the dataset used to build this Vocab, but really the user wants it to be mapped to index "0" (which is what Vocab currrently does).