Add Keras.io Example : `Token Classification using RoBERTa`

[kanpuriyanawab]

In favor of #189

This huggingface tutorial can be referred !

cc : @jbischof @mattdangerw , Do you think it's required ?

[jbischof]

@shivance I think text classification we have covered in spades with https://keras.io/guides/keras_nlp/getting_started/. However, almost any other fine-tuning task beside sentence similarity would be a non-trivial extension from our guide. Any of those interesting to you?

[kanpuriyanawab]

@jbischof How about token classification task?

[jbischof]

That would be very interesting, thank you!

[mattdangerw]

@shivance yeah a token classification guide would be great! Could be any encoder model you could like (RoBERTa sounds good).

One option for a task would be the conll named entity recognition dataset. https://www.tensorflow.org/datasets/catalog/conll2003

[kanpuriyanawab]

Sure, Thanks for the input @mattdangerw .

[kanpuriyanawab]

An additional question, for keras-io example, the PR should be opened in keras-io repo right? and at examples\nlp location?

[mattdangerw]

Yes and yes! Feel free to ask any questions here though. We can use this for planning.

[kanpuriyanawab]

I was referring [Huggingface guide](https://huggingface.co/course/chapter7/2 for token classification.
As model adds extra tokens and does subword tokenization for this task, we need to align the labels with the tokens

They used following function for that

Huggingface Fast Tokenizer has method word_ids() which allows them to find the parent word of a subword token.

Do we have something to tackle this problem in keras-nlp, easily and effectively. I tried to take a deeper look into implementation of word_ids() method, but found that it's just a python binding over rust implementation (HF tokenizers implemented in Rust).

https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/tokenizer/encoding.rs
this is the file with class Encoding which is supposed to hold implementation of word_ids() , but I didn't find any lead.

Can you help?

[kanpuriyanawab]

Right now my approach is 1. trying to get parent word Id's for each token, and 2. Re use huggingface's function to align, which uses parent word ids

I am using a brute force (kindof), approach to get the parent word (later can easily be converted to IDs).

Code

import re

def split(sentence):
  x = re.split(r"([^a-zA-Z0-9])", sentence)
  x = [ele for ele in x if ele != ' ' and ele !='']
  return x

def removeG(word):
  if word.startswith('Ġ'):
    return word[1:]
  return word

def find_parent_(preprocessor, sentence):

  tokens = preprocessor.tokenizer(sentence).numpy()
  sentence = split(sentence)
  #print(sentence)
  
  parent = []

  j = 0

  for i, word in enumerate(sentence):  
    tmp = ""
    tmp_par = []

    while j<len(tokens):
      #print("j = ", j)
      token = preprocessor.tokenizer.id_to_token(tokens[j])
      #print("token = ", token)
      tmp += removeG(token)
      tmp_par.append(word)

      j+=1
      #print("tmp = ", tmp, "word = ", word)
      if tmp == word:
        parent = [*parent, *tmp_par]
        #print("parent = ", parent)

        break

  return parent

For sentence = ['Thailand', "'", 's', 'powerful', 'military', 'thinks', 'the', 'government', 'is', 'dishonest', 'and', 'Prime', 'Minister', 'Banharn', 'Silpa', '-', 'archa', "'", 's', 'resignation']
it returns parent = ['Thailand', 'Thailand', "'", 's', 'powerful', 'military', 'thinks', 'the', 'government', 'is', 'dishonest', 'and', 'Prime', 'Minister', 'Banharn', 'Banharn', 'Banharn', 'Silpa', 'Silpa', '-', 'archa', 'archa', "'", 's', 'resignation']

Doesn't look correct. Can we do it in some better way and How can we do it more efficiently?

cc: @mattdangerw @jbischof @abheesht17

[kanpuriyanawab]

Now that I took a deeper look, Fairseq also used similar methodology to mine ( can be found in this colab )

https://github.com/facebookresearch/fairseq/blob/ad0e69cd99e1ff884041fbd8467d1404bd09847a/fairseq/models/roberta/alignment_utils.py#L12-L97

[TheAthleticCoder]

Hey!
I was wondering if we could use labelling spans which refer to the lengths of text in the input sequence that are assigned a label. Instead of mapping the subword tokens to the words in the sentence, we could map the subword tokens to the label spans instead.
Suppose:

Sentence: 'The Happy Cat'
Label Spans: [(0,2),(4,8),(10,12)]
Labels:[X,Y,Z] (would be the same length as the Label Spans)
Subtokens: ['GThe','GHa','Gppy','GCat']

The first loop could be through all the Subtokens and the second loop through the Label Spans. Then we could find the spans that are overlapping with the current token and assign the label to that token.

[mattdangerw]

@shivance good question! At the highest level, probably have the same feedback here as #741 (comment) (squad is also a token classification problem basically).

Let's just do our preprocessing, including calculation of the labels, in python for now, in the simplest way we can. Let's focus on simple readable code, especially as this will end up as a keras.io guide. After the preprocessing is done, we can make sure training is still efficient by making sure the dataset is cached and not recomputed every epoch. Some sort of simple python routines to look for words sounds goods to me.

Some useful APIs we have today... tokenizer.get_vocabulary() will get you the the full list of tokens. And you can pass dtype="string" to a tokenizer, e.g. tokenizer = keras_nlp.models.RobertaTokenizer.from_preset("roberta_base_en", dtype="string") to get string subword token output.

Long term, we will definitely want some utilities for doing this, and we will want to be able to run all our preprocessing inside a TF graph. But let's start with a simple example built on top of our existing APIs, and use that to inform future improvements.

[kanpuriyanawab]

This work is still WIP.