Skip to content

Commit b9b0f58

Browse files
committed
Resolve rebase change
1 parent 191ab03 commit b9b0f58

File tree

1 file changed

+1
-1
lines changed
  • src/Microsoft.ML.Tokenizers/PreTokenizer

1 file changed

+1
-1
lines changed

src/Microsoft.ML.Tokenizers/PreTokenizer/Roberta.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ public override IEnumerable<Split> PreTokenize(string text, bool considerSpecial
3030
return Array.Empty<Split>();
3131
}
3232

33-
return SplitText(sentence, Tokenizer.P50kBaseRegex());
33+
return SplitText(text, Tokenizer.P50kBaseRegex());
3434
}
3535
}
3636
}

0 commit comments

Comments
 (0)