pyonmttok
Fast and customizable text tokenization library with BPE and SentencePiece support
Installation
In a virtualenv (see these instructions if you need to create one):
pip3 install pyonmttok
Dependencies
- None
Releases
| Version | Released | Bookworm Python 3.11 |
Trixie Python 3.13 |
Files |
|---|---|---|---|---|
| 1.38.1 | 2026-01-10 | |||
|
|
||||
| 1.38.0 | 2025-12-31 | |||
|
|
||||
| 1.37.1 | 2023-03-01 | |||
|
|
||||
| 1.37.0 | 2023-02-28 | |||
|
|
||||
| 1.36.0 | 2023-01-11 | |||
|
|
||||
| 1.35.0 | 2022-12-06 | |||
|
|
||||
| 1.34.0 | 2022-09-13 | |||
|
|
||||
| 1.33.0 | 2022-08-29 | |||
|
|
||||
| 1.32.0 | 2022-07-25 | |||
|
|
||||
| 1.31.0 | 2022-03-07 | |||
|
|
||||
| 1.30.1 | 2022-01-25 | |||
|
|
||||
| 1.30.0 | 2021-11-29 | |||
|
|
||||
| 1.29.0 | 2021-10-08 | |||
|
|
||||
| 1.28.1 | 2021-09-30 | |||
|
|
||||
| 1.28.0 | 2021-09-17 | |||
|
|
||||
| 1.27.0 | 2021-08-30 | |||
|
|
||||
| 1.26.4 | 2021-06-25 | |||
|
|
||||
| 1.26.3 | 2021-06-24 | |||
|
|
||||
| 1.26.2 | 2021-06-08 | |||
|
|
||||
| 1.26.1 | 2021-05-31 | |||
|
|
||||
| 1.26.0 | 2021-04-19 | |||
|
|
||||
| 1.25.0 | 2021-03-15 | |||
|
|
||||
| 1.24.0 | 2021-02-16 | |||
|
|
||||
| 1.23.0 | 2020-12-30 | |||
|
|
||||
| 1.22.2 | 2020-11-12 | |||
|
|
||||
| 1.22.1 | 2020-10-30 | |||
|
|
||||
| 1.22.0 | 2020-10-29 | |||
|
|
||||
| 1.21.0 | 2020-10-22 | |||
|
|
||||
| 1.20.0 | 2020-09-24 | |||
|
|
||||
| 1.19.0 | 2020-09-02 | |||
|
|
||||
| 1.18.5 | 2020-07-07 | |||
|
|
||||
| 1.18.4 | 2020-05-22 | |||
|
|
||||
| 1.18.3 | 2020-03-09 | |||
|
|
||||
| 1.18.2 | 2020-02-17 | |||
|
|
||||
| 1.18.1 | 2020-01-16 | |||
|
|
||||
| 1.18.0 | 2020-01-06 | |||
|
|
||||
| 1.17.2 | 2019-12-06 | |||
|
|
||||
| 1.17.1 | 2019-11-28 | |||
|
|
||||
| 1.17.0 | 2019-11-13 | |||
|
|
||||
| 1.16.1 | 2019-10-21 | |||
|
|
||||
| 1.16.0 | 2019-10-07 | |||
|
|
||||
| 1.15.7 | 2019-09-20 | |||
|
|
||||
| 1.15.6 | 2019-09-19 | |||
|
|
||||
| 1.15.5 | 2019-09-16 | |||
|
|
||||
| 1.15.4 | 2019-09-14 | |||
|
|
||||
| 1.15.3 | 2019-09-13 | |||
|
|
||||
| 1.15.2 | 2019-09-11 | |||
|
|
||||
| 1.15.1 | 2019-09-05 | |||
|
|
||||
| 1.15.0 | 2019-09-05 | |||
|
|
||||
| 1.14.1 | 2019-08-07 | |||
|
|
||||
| 1.14.0 | 2019-07-19 | |||
|
|
||||
| 1.13.0 | 2019-06-12 | |||
|
|
||||
| 1.12.1 | 2019-05-27 | |||
|
|
||||
| 1.11.0 | 2019-02-05 | |||
|
|
||||
| 1.10.6 | 2019-01-15 | |||
|
|
||||
| 1.10.5 | 2019-01-04 | |||
|
|
||||
| 1.10.4 | 2018-12-18 | |||
|
|
||||
| 1.10.1 | 2018-10-09 | |||
|
|
||||
| 1.10.0 | 2018-10-05 | |||
|
|
||||
| 1.8.4 | 2018-09-24 | |||
|
|
||||
| 1.7.0 | 2018-09-04 | |||
|
|
||||
| 1.6.2 | 2018-09-03 | |||
|
|
||||
| 1.6.1 | 2018-07-31 | |||
|
|
||||
| 1.5.3 | 2018-07-13 | |||
|
|
||||
| 1.5.2 | 2018-07-12 | |||
|
|
||||
| 1.5.0 | 2018-07-03 | |||
|
|
||||
| 1.4.0 | 2018-06-13 | |||
|
|
||||
| 1.3.0 | 2018-04-07 | |||
|
|
||||
Issues with this package?
- Search issues for this package
- Package or version missing? Open a new issue
- Something else? Open a new issue