Memory overflow using dict type features

[napsternxg]

I am trying to use Word vector features for training my CRF model for Named entity recognition.I am using sklear-crfsuite for training my model which follows the same convention for creating features as this library and is a wrapper around python-crfsuite.

The word vector features result in 300 additional dense features per token in the sequence. Because python-crfsuite uses dictionaries to specify the features, my training data itself ends up taking 6 times more memory compared to that used by CRFsuite native binary with the training data in text files. (35 GB vs 6.5 GB).

Is there a more memory efficient way to specify the features instead of only using dicts. I believe the ItemSequence is a way to pass Sequences instead of using the list of dictionaries. Can I also pass generators for my training features instead of the list of training features, to keep the memory overhead low ?

I know sklearn-crfsuite uses the six.moves.zip function which returns an iterator instead of a list of zipped X,y, so I believe it should be possible.

Can there be an example for using the ItemSequence class for giving the input data?

[kmike]

Hey @napsternxg,

Yes, generators are supported, you don't have to keep all dictionaries in memory at the same time.
It looks like you don't need ItemSequence either - just call Trainer.append in a loop, generating dicts on fly.

[foxinfotech]

Hey @napsternxg

Did you solve your issue? I am also having the same issue, if you have solved it can you please share the code.

[kmike]

@foxinfotech what have you tried? The code should be really straightforward, I'm not sure what the issue is:

data = [...]  # a list with raw examples
y = [...]

def get_X(data):
    for seq in data:
        yield [extract_features(item) for item in seq]

trainer = pycrfsuite.Trainer()

for xseq, yseq in zip(get_X(data), y):
    trainer.append(xseq, yseq)

[foxinfotech]

@kmike :My exact issues is:
I have sentences represented in arrays, where each word is represented in rows (300 dimensions). I need to pre-process the sequence one by one using a neural network node which expands the word vectors to 1000 dimensions.
For each processed sentence by the NN, I am collecting the outputs of neural network in list where each array has words in rows but this time the columns increases from 300 to 1000. this pushes me to memory overflow. I am looking for something which makes the CRF learning online.
I am a Java guy and new to python, please excuse me if you find this question silly.

[kmike]

@foxinfotech aha, I see, so the problem is not with RAM used by dict features, but in that you don't want to load the training set to RAM at once at all.

There are online training algorithms in crfsuite (e.g. SGD), but AFAIK it doesn't expose an interface to train on mini-batches, at least not in C++ API. It should be possible to fix in https://github.com/chokkan/crfsuite library.

[napsternxg]

@foxinfotech yes I did resolve the issue. I used the following strategy:

def get_line(filename):
    with open(filename, "wb+") as fp:
        seq = []
        for line in fp:
            line = line.strip()
            if not line:
                yield seq
                seq = []
            line = line.split("\t")
            seq.append(line)

def get_features(seq, i):
    /** DEFINE EXTRACTED FEATURES **/
    return features ## a dict

LBL_IDX = -1 ## Index of label
X = ([get_features(seq, i) for i in xrange(len(seq)) for seq in get_line(filename)])
Y = ([seq[i][LBL_IDX] for i in xrange(len(seq)) for seq in get_line(filename)])

trainer = pycrfsuite.Trainer()
trainer.fit(X, Y)

Eventually, the trainer is still going to load the final data into its memory. In my case this was 5-6 GB with 300 dimensional word embedding and 10,000 sequences. if your machine doesn't have the required RAM then CRFSuite will not be able to fit this data.

[foxinfotech]

def get_features(seq, i):
/** DEFINE EXTRACTED FEATURES **/
return features ## a dict

for eg. my sentence is "i ate apple", i have vector representation say 4 dims for each word is available. How should the feature dict for each word look like??

for word 'ate':
{'f1':2.0,'f2':3.0,'f3':2.3,'f4:'2.1 }

Am i right on this?

[kmike]

@foxinfotech yes, exactly.

It may be better to add a direct support for this kind of features, so that you won't have to create these dicts - it should work faster. It can be implemented as an additional input format for ItemSequence (see https://github.com/tpeng/python-crfsuite/blob/720334fc325534e5b56f2b83ed023645ac9bd0d1/pycrfsuite/_pycrfsuite.pyx#L39). I think it may accept dicts with np.array, array.array values or lists of floats; the API could look like {'f': [2.0, 3.0, 2.3, 2.1]}, which will be expanded to {'f0':2.0, 'f1':3.0, 'f2':2.3, 'f3:'2.1 } internaly.

Pull requests are welcome :)

[napsternxg]

@tpeng, @kmike and @foxinfotech I have made a pull request with an API change which allows for easy addition of word embedding or other float list features. Do have a look and let me know if this is compatible with the overall API of python-crfsuite.

[napsternxg]

I have created a separate issue for the word embedding feature at #39. Please continue the discussion on it at that post. I am closing this issue as it is resolved.