tokenizer-tools
Tools for tokenizer develope and evaluation
Installation
In a virtualenv (see these instructions if you need to create one):
pip3 install tokenizer-tools
Dependencies
Releases
| Version | Released | Bookworm Python 3.11 |
Trixie Python 3.13 |
Files |
|---|---|---|---|---|
| 0.46.1 | 2020-09-22 | |||
| 0.46.0 | 2020-09-04 | |||
| 0.45.0 | 2020-09-04 | |||
| 0.44.1 | 2020-09-22 | |||
| 0.44.0 | 2020-08-10 | |||
| 0.43.0 | 2020-08-07 | |||
| 0.42.0 | 2020-08-07 | |||
| 0.41.0 | 2020-07-27 | |||
| 0.40.0 | 2020-07-17 | |||
| 0.39.0 | 2020-07-17 | |||
| 0.38.0 | 2020-06-22 | |||
| 0.37.0 | 2020-05-18 | |||
| 0.36.2 | 2020-05-13 | |||
| 0.36.1 | 2020-05-13 | |||
| 0.36.0 | 2020-05-07 | |||
| 0.35.0 | 2020-05-04 | |||
| 0.34.0 | 2020-04-27 | |||
| 0.33.2 | 2020-04-16 | |||
| 0.33.1 | 2020-04-16 | |||
| 0.33.0 | 2020-04-16 | |||
| 0.32.0 | 2020-04-16 | |||
| 0.31.0 | 2020-04-14 | |||
| 0.30.0 | 2020-04-10 | |||
| 0.29.0 | 2020-04-07 | |||
| 0.28.0 | 2020-04-04 | |||
| 0.27.0 | 2020-04-03 | |||
| 0.26.0 | 2020-04-02 | |||
| 0.25.0 | 2020-03-31 | |||
| 0.24.0 | 2020-03-30 | |||
| 0.23.0 | 2020-01-21 | |||
| 0.22.2 | 2020-01-20 | |||
| 0.22.1 | 2020-01-20 | |||
| 0.21.1 | 2020-01-14 | |||
| 0.21.0 | 2020-01-14 | |||
| 0.20.0 | 2019-12-06 | |||
| 0.19.0 | 2019-12-05 | |||
| 0.18.0 | 2019-12-05 | |||
| 0.17.1 | 2019-11-22 | |||
| 0.16.0 | 2019-11-12 | |||
| 0.15.7 | 2019-09-29 | |||
| 0.15.6 | 2019-09-29 | |||
| 0.15.3 | 2019-09-18 | |||
| 0.15.2 | 2019-09-17 | |||
| 0.15.0 | 2019-09-09 | |||
| 0.14.1 | 2019-06-28 | |||
| 0.14.0 | 2019-05-20 | |||
| 0.13.1 | 2019-03-29 | |||
| 0.13.0 | 2019-03-29 | |||
| 0.12.0 | 2019-03-11 | |||
| 0.11.0 | 2019-01-15 | |||
| 0.10.0 | 2018-11-21 | |||
| 0.9.0 | 2018-11-16 | |||
| 0.8.2 | 2018-10-18 | |||
| 0.8.1 | 2018-10-18 | |||
| 0.8.0 | 2018-10-18 | |||
| 0.7.1 | 2018-09-19 | |||
| 0.7.0 | 2018-09-19 | |||
| 0.6.0 | 2018-09-13 | |||
| 0.5.0 | 2018-09-11 | |||
| 0.4.2 | 2018-09-10 | |||
| 0.4.1 | 2018-09-10 | |||
| 0.4.0 | 2018-09-10 | |||
| 0.3.2 | 2018-09-07 | |||
| 0.3.1 | 2018-09-06 | |||
| 0.3.0 | 2018-09-06 | |||
| 0.2.2 | 2018-09-06 | |||
| 0.2.1 | 2018-09-06 | |||
| 0.1.5 | 2018-09-06 | |||
Issues with this package?
- Search issues for this package
- Package or version missing? Open a new issue
- Something else? Open a new issue