Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vocabulary file handling #57

Open
mh-northlander opened this issue Mar 10, 2023 · 0 comments
Open

Vocabulary file handling #57

mh-northlander opened this issue Mar 10, 2023 · 0 comments

Comments

@mh-northlander
Copy link
Collaborator

JapaneseWordPieceTokenizer which we use to build the vocabulary recognizes '\n' (or ' ') as a token.
BertSudachipyTokenizer however removes them from the tokenization results.
Currently we just ignore those tokens (and problems caused by that (#54)).

  1. We may need some error handling on the vocab file corruption.

  2. It maybe better to make those tokens used.
    In this case we need to prepare a new vocab file format (current txt format cannot handle '\n').
    We also need to modify chiTra tokenizer, and reconsider the corpus cleaning processes relating to those tokens.

  3. In the case we do not use those tokens, we should remove them during vocab building.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant