MosesTokenizer has been moved out of NLTK due to licensing issues

[alvations]

@jekbradbury great work here!

Due to nltk/nltk#2000, we had to remove MosesTokenizer out of NLTK but now it's hosted on https://github.com/alvations/sacremoses

pip install sacremoses

The silver lining is that the package comes with the data needed for tokenization so there's no need to keep the nltk_data directory =)

---

I would propose adding sacremoses on top of nltk because NLTK has another port of a nice tokenizer (by @jonsafari) that people overlook, https://github.com/nltk/nltk/blob/develop/nltk/tokenize/toktok.py (I think it's fast too)

[alvations]

I'm not sure which branch is the dev branch to add tokenizers, so I'll rely on the github search:

text/torchtext/data/utils.py

    elif tokenizer == "moses":
        try:
            from sacremoses import MosesTokenizer
            moses_tokenizer = MosesTokenizer()
            return moses_tokenizer.tokenize
        except ImportError:
            print("Please install SacreMoses. "
                  "See the docs at https://github.com/alvations/sacremoses for more information.")
            raise

    elif tokenizer == "toktok":
        try:
            from nltk.tokenize.toktok import ToktokTokenizer
            toktok = ToktokTokenizer()
            return toktok.tokenize
        except ImportError:
            print("Please install NLTK. "
                  "See the docs at https://nltk.org  for more information.")
            raise

Hope it helps!

[PetrochukM]

@alvations Thanks for the repo! We're coming up on the same problem: https://github.com/PetrochukM/PyTorch-NLP

I'll update it to SacreMoses.