Hugging Face's logo

Spaces:
xu-song
/
tokenizer-arena
Running

App Files Community
1

Is there any documentation within this leaderboard?

#1
by zhiminy - opened
Discussion
zhiminy
•
edited Jun 29, 2024

I cannot locate any specification in this space...What is this leaderboard used for?
xu-song
 Owner

Thanks for your attention. A brief document has been added to the demo.
The leaderboard aims to evaluate tokenizer performance on different languages.

  • Lower oov_ratio refers to less out-of-vocabulary tokens.
  • Higher char/token means less words be segmented into subwords.
xu-song changed discussion status to closed

Sign up or log in to comment