Embed the Tokenizer data files inside the assembly

[tarekgh]

Fixes #6368

[tarekgh]

@michaelgsharp could you please have a look and let me know if you have any questions? Thanks!

[codecov]

Codecov Report

Merging #6403 (eb085c6) into main (1903fa5) will decrease coverage by 0.05%.
The diff coverage is 68.18%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6403      +/-   ##
==========================================
- Coverage   68.53%   68.48%   -0.06%     
==========================================
  Files        1169     1169              
  Lines      246863   246867       +4     
  Branches    25782    25788       +6     
==========================================
- Hits       169198   169070     -128     
- Misses      70948    71052     +104     
- Partials     6717     6745      +28     

Flag	Coverage Δ
Debug	68.48% <68.18%> (-0.06%)	⬇️
production	62.88% <65.57%> (-0.07%)	⬇️
test	88.94% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/Microsoft.ML.TorchSharp/Utils/FileUtils.cs	0.00% <ø> (-24.62%)	⬇️
test/Microsoft.ML.Tests/TextClassificationTests.cs	94.83% <ø> (ø)
...rc/Microsoft.ML.Tokenizers/Model/EnglishRoberta.cs	78.53% <62.50%> (-2.51%)	⬇️
...ft.ML.TorchSharp/Extensions/TokenizerExtensions.cs	87.50% <100.00%> (-0.97%)	⬇️
...crosoft.ML.Tokenizers.Tests/EnglishRobertaTests.cs	100.00% <100.00%> (ø)
...osoft.ML.KMeansClustering/KMeansPlusPlusTrainer.cs	83.92% <0.00%> (-7.16%)	⬇️
src/Microsoft.ML.Data/Training/TrainerUtils.cs	66.26% <0.00%> (-3.82%)	⬇️
...crosoft.ML.StandardTrainers/Standard/SdcaBinary.cs	85.40% <0.00%> (-3.42%)	⬇️
...crosoft.ML.Data/DataLoadSave/Binary/BinarySaver.cs	87.75% <0.00%> (-2.83%)	⬇️
src/Microsoft.ML.Sweeper/AsyncSweeper.cs	71.42% <0.00%> (-1.37%)	⬇️
... and 6 more

[tarekgh]

@michaelgsharp any idea about this failure? This is only on netfx461 only.

    Microsoft.ML.Tests.TextClassificationTests.TestSingleSentence3Classes [FAIL]
      Assert.Equal() Failure
                      � (pos 6)
      Expected: Class Three
      Actual:   Class One
                      � (pos 6)
      Stack Trace:
        D:\a\_work\1\s\test\Microsoft.ML.Tests\TextClassificationTests.cs(249,0): at Microsoft.ML.Tests.TextClassificationTests.TestSingleSentence3Classes()
      Output:
        Test TestSingleSentence3Classes: aborted: passed

[michaelgsharp]

@tarekgh with the small amount of training we do for the TorchSharp tests, they sometimes do that. I have started changing the tests that fail frequently to not care about what class is returned for those short tests.