Skip to content
This repository was archived by the owner on Sep 10, 2025. It is now read-only.

Conversation

@reachsumit
Copy link
Contributor

Description

Avoid using std::regex and fix lint errors

This change fixes the lint errors raised from merging #1916 into fbcode.

Types of changes

[x ] Fixes

Changes made

  • Removed std::regex usage and opted to use re2
  • Fixed indexing in pre-tokenization step
  • Addressed additional linter issue identified internally

Testing

  • No issue identified in pre-commit
  • No issue identified with any of the unit tests.

Copy link
Contributor

@Nayef211 Nayef211 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for fixing these errors

Copy link
Contributor

@abhinavarora abhinavarora left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me!

@abhinavarora
Copy link
Contributor

I'll go ahead and merge these changes as the unit test failures seem completely unrelated to this PR.

Copy link
Contributor

@abhinavarora abhinavarora left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@abhinavarora abhinavarora merged commit 6ffe7be into pytorch:main Oct 10, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants